0:00If you think fable concerns are bad, like see when Whimo hits a dog nest as if people lose their mind. Imagine when first robot knocks off a toddler off a kitchen table, you're going to see some real strict liability. So physical AI, the level of stringency just goes up and up and up and up. Um, so that's kind of like the big picture, agents, models, robotics. As the technology progresses, as agents get longer horizons, new types of failure modes will emerge that will
0:28also bring in just new kinds of ways to create value, but also more risk surface. You'll start to see true agent to agent interactions that are not mediated by humans. There's going to be a bunch of interesting questions. You're basically going to need a new legal system. How do they build trust amongst each other?
0:44Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content.
0:56We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way.
1:07But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you we'll never stop working to make the show even better. Now, let's get into it.
1:32Okay, we're in the studio with Run from AI, AI underwriting company, uh, with our trusty co-host, Vivu. Welcome.
1:38Thank you. Thanks for having me. Thank you.
1:40Uh, what are you announcing today? We have raised $40 million led by Ribbit Capital and Firstmonic.
1:46You first came to my attention when uh Nat and Dan invested in you guys. Is the story like pretty much the same? Like we are you today where you thought you were back then?
1:54When we raised our seed round, we had a hypothesis that at some point risk was going to hold down adoption. At that point in time that felt kind of hypothetical and I think that that is now over. Clearly the moment is now with uh mythos and with fable it's pretty obvious that literally the binding constraint on adoption is risk and so for us it feels like this is a natural continuation of the same hypothesis but where previously was speculation now it feels like fact and let's get the list of the customers
2:24that you um highlighting as part of your series a totally yeah so we are now working with folks like cursor Harvey lovable 11 labs yeah amazing congrats thank you so you were famously one of the first like the first uh hired and and topic for GTM and product. I'm just kind of curious like what was your path into AI? Just recap.
2:48Late 2021 I sold a company, my first company, an edtech company. I had a bit of time to think about what was next. I came across a scaling wallpaper and that just struck me like lightning. I was like this is a big idea. In short, the scale paper just says the bigger the model, the smarter the model. And this is the Kaplan one, not the chinchilla one.
3:09Exactly the Kaplan one. And the important thing that clicked for me there was, oh, now capital will understand this. If you put in more money, you get more money out. And so that will kick off a hype cycle. Uh, and so you'll actually kind of you'll get a sense of predictable returns, which is in fact what's played out. And so I just packed my bags. I'd never been to San Francisco. I just packed my bags throughout here to find the people who ridden it. Uh and at the time they had
3:37just started a small lab colanthropic.
3:39There was like 40 people at the time or so. Drank a bunch of coffee until I eventually got introduced to Dario. And at the time they were wrestling with some of these questions of like should we deploy our models? Should we make revenue? How should we engage with the rest of the world? They just broken off from OpenAI. Uh and it's been publicly reported that they were kind of concerned with how they were dealing with deployment. So they were wrestling with some of those questions at that this point. This is like early fog of war, like early 2022. The sexiest
4:08product at the time was like Jasper, like there's there's nothing out there. So, where is value going to crew? Uh what are going to be the different parts of the stack were all open questions.
4:16I want to highlight to people, you ask these questions because you have a PPE background.
4:21Uh I actually was in in Singapore in one of the sort of feeder programs for prepping people for PPE. So, I had a tutor. We learned uh you know, philosophy and politics and economics. But like I think your kind of like machine learning people who read the neural uh scaling laws paper would not necessarily draw the same conclusions that you did whereas any capitalist would read that and go holy Correct.
4:49Right. Like yes uh who tipped you onto that paper because it's not a paper that you normally read, right? Like in your circles.
4:55Yeah. I think I'd actually uh ever since Alph Go had had some appreciation that AI was a big deal. Uh but it kind of felt it raised all these kind of interesting philos philosophical questions, but it was kind of not clear from afar where exactly that would go.
5:14But it was obvious enough that it was like this is going to be a big thing if we find the kind of right mechanism to kind of get the technoc capital machine to work on this. but it was just not clear. And so I think it was some way in which like that became obvious and also it wasn't as obvious at the time than than it is now, right? Like it was just like wow this is so interesting but I still felt coming from kind of a philosophy and economics background it felt like if this turns out to be true you're going to be wrestling with all of
5:43the big questions in society. Everything you've learned about politics gets thrown out of the window. Everything you've learned about economics at least gets challenged. And so what felt interesting was to be at that frontier that has just ramifications across everything. So that's that's why I I I thought it sorted out.
6:01I mean clearly really good insight for people people who don't know PP the PP program is like where prime ministers are born. So then you end up meeting Jared.
6:10Yep. Uh first Dario. Yeah.
6:12Yeah. Uh well I mean like so did you get extra insights from talking with them that you didn't get from your original hypothesis? If you read the scale paper, you get this like very vague sketch of like, wow, this seems kind of important.
6:24There are some lines on a chart. This seems kind of important. Um, and what I think the team anthropic has thought more about than anyone was like, what are the implications of this if you really play this out? Uh, and back then they had kind of vision documents for what the world would look like in 2026.
6:39And there are kind of in vivid detail playing out how much comput is going to be needed, what is the capex going to look like, what are going to be some of the kind of societal concerns, but also what is the amount of economic value coming out here. And so it kind of felt like they held a crystal ball that in hindsight just be dramatically correct.
7:00And they weren't holding it like they were obviously correct. They were just like take this hypothesis really really seriously.
7:06Think it through and think it through. in the same way the kind of situational awareness that is now across the street. Yeah.
7:14Oh my god, we're all in the same one square mile of right and that's now a couple years old but also people keep referencing it these particular weeks with Fable and Methos and it's like wow if you take this one idea seriously the scale in the north a lot of things fall into place.
7:32And keep in mind at this point this is the same team that had did GPT 1 2 and 3 which is also like it's not just some experimentation like there this is a real model that that we just scaled up and they had deep conviction in in again in this like big if you take a big blob of compute data it just wants to learn and out of that will come smart and smart models and all the particulars were not clear.
7:55Yeah. Yeah. and all the implications were not clear but that deep conviction is this like core thesis and that was kind of dizzying is both phenomenally interesting and exciting and also very quickly you got to like the world we know today will no longer be this is this hypothesis holds so it also just felt like important in some kind of grand sense what kind of shaped you there so that was your early 2022 not only had GPT123 come out but you know the amazing
8:24co-founders of anthrop IC that have never split up. The only ones they actually had the conviction to leave OpenAI, start their lab. You said there were about 40 people there. What was the time like there?
8:35It was kind of remarkably like what it looks like on the outside today.
8:39Extremely cohesive, extremely missionoriented and living in this tension between their two ideas which is AI could both go really well and really bad and we want to be part of building it. that creates astounding amounts of tension and they were wrestling with this incentive challenge where they know they're kind of there's a race that they're in where you might get forced to cut corners but it also felt very important to them to
9:06be at the forefront of technology and all of those ideas were just present at that time. It kind of feels like that line has been just very very clear and I think kind of love them or hate them they have really stuck to their guns. is there's a core set of beliefs that they hold more deeply than most companies hold any beliefs.
9:27Yeah. Fast forward to today. What does that lead us to AI underwriting company?
9:31What are you up to? What what motivated you to start this?
9:34Yeah. AI built confidence infrastructure for Frontier AI through standards and insurance. The link from Enthropic to building confidence infrastructure.
9:44Looking out the windows at anthropic offices and seeing Whimos driving by already back then early 2022 Whimos were in some ways like AGI for cars. like there were superhuman drivers, but you couldn't take one to the airport. And now, 400 years later, you still can't take a Whimo to the airport, despite now everyone having kind of looked at the evidence and being like, they're better drivers than humans. So, in that particular instance, what's clear is that the binding constraint on AI being useful is not capability, but instead
10:11liability or risk or trust. That problem is uh general. The reason why right now Fable is not open for access is not because it's not a good model. It's because it's a very good model. It's just hard to make promises about what it will will not do. And this problem gets worse as AI gets better. Basically, more intelligent AI can be more autonomous.
10:36That's more valuable, but also the risk surface grows. And so the what Whimo illustrates is that unless you build the conference infrastructure to make promises about AI or at least bring light to the risks, you grind adoption to halt. Governments, banks, hospitals, militaries need to have some sense of what AI will and will not do to be able to operate for them to incorporate it.
11:01And that's the problem that we're trying to solve. Now why standards and insurance? Uh if you trace this problem back through history, every technology wave has had some version of this problem. So if you go back to like year 1900, electricity comes out.
11:16Franklin, cars burn down, sorry, houses burn down, lots of people die. 1930s, cars are a big deal, kill lots of people. 50s, private nuclear energy is a big deal, poses big risks. In each of those instances, the market runs ahead of regulation to create confidence infrastructure because that's required to make go decision. They're required for adoption and the market fundamentally wants adoption. And in all of those instances, common blueprint
11:45emerges between standards and insurance. The reason it's these two components is standards kind of provide the rules of the road and they also specify like what are the tests that need to be run so we can get a sense of how high the risk is.
11:57Taking the case of cars, it's like a car crash. Great. Everyone, they inform your insurance pricing today. They inform your purchasing decisions, etc. That's basically the risk framework. The insurers are important because they pick up the bill. So, they are the private institution that is most on the side of best incentivized to quantify the risk truthfully and then figure out all the ways to reduce the risk because that increases their profit. So they're basically they help shape the incentives and these two worked really well in
12:27unison. Now how does that show up as a company? Well, one of the things that was obvious even or starting to become obvious even a couple years ago was that frontier companies some of our customers today like Kursar uh Sierra 11 Labs Javi were going to have a very easy time selling a pilot to a bank. Like the demo just sells itself. It's magic. But bringing that through if you want to do a wall-to-wall roll out at a bank or a hospital, uh you have to go through the
12:57risk process. These banks have no idea even which questions to ask, let alone which answers are sufficient, let alone like how do they go and test whether these agents actually work the way they're supposed to. And so they they had this problem of like what can we say to earn the trust? And we think there's like a golden sentence that goes something like, "Hey, I hear you're really worried about hallucinations or jailbreaks or whatever it may be. We've had an independent third party test us
13:25against the gold standard. We pass the flying colors and as a vote of confidence, the world's most conservative insurers have looked at the data and are willing to take some of the risk onto their balance sheet." Yeah.
13:36So if something does go wrong, there's money behind it. Yeah.
13:38Exactly. That's kind of like the link between all of them. We can we can get into some of the the hard parts related to the technical testing which is I think the crux of the matter. Uh but I'll pause there.
13:48How did you and Rejie come together?
13:50This there's always like you you come across very confident and you know you we're announcing your series A and all these things but I want to see like the early initial stages of like idea formation.
14:00Yeah. Rajiv is actually my soon to be brother-in-law.
14:04Oh. So I'm actually uh in a week and a half getting married to Raj's sister.
14:11Okay, now you're tight.
14:13You know, so Rajiv and I have known each other for a decade. Uh funny story, I met both Rajiv and his sister Henna uh at the same time when Hannah and I were interns at McKenzie in London and Rajiv was assigned as my mentor. Uh so met them at the same time. For the longest time it was not obvious that we were necessarily going to work together. Uh I was in startups. He was an insurance partner at McKenzie three or four years ago. I think Hannah convinced him that AI was going to be a really big saying
14:43and so he quit his job cushy partner job at Mackenzie in London. Pack his bags through San Francisco and ended up joining meter. You guys are probably enough. Exactly. We see the the chart of uh the horizons of the task that agents can take on is is doubling extremely fast. So here zero there led their partnerships with anthropic and openai to test their models before release but also working closely with the US and UK
15:11government uh to figure out like how do you know whether a model can be released and in some ways that's like the perfect background. He's spent a lot of time in insurance, knows that world, spent a lot of time with frontier testing of models.
15:26And so when I was bumbling around this idea space, starting with some of the ideas we talked about related to Whimo, as soon as we got into the content, we were both like, "Oh, this would be an amazing business to to build together.
15:37This is like wrestling with the problem that we both think is the most important in the world." From a market angle, which is kind of our intuitions is that the market can do a lot. And the faster AI moves, the harder it is for government to solve some of these problems. And then it took a little bit of time to work through what is it like to work with family.
15:56Uh and uh cuz you're already dating at the time or Yeah. Yeah.
16:02Exactly. Already back then it was we felt like we're a family and so starting a business together felt like kind of a big step and uh here we are with just immense amounts of trust.
16:12Yeah. So now you're a company of how big? How big you guys?
16:15There is just 20 of us now. Tony, if you guys now have series A and you have your first certification out, the AIU1. Um, let's bring up the certification.
16:24So, this is the agent certification, right? What goes into the process? I have like two questions here. One is walk us through the certification and two is what is the process for a company to get certified? You know, great. As it says right at the top, A1 is a standard for agent security, safety, and reliability. The fundamental design principle is take all of the concerns that slow down adoption. So all the questions, all the fears that keep uh security leaders in the Fortune 1000
16:54up at night and put them into one comparison framework. Uh that's what you'll see there. You can see the six categories. Two, you want to ground all of this in technical testing. So one of the concerns with security standards that often feel kind of like theater paperwork is that they don't actually ground out in does any of this work?
17:11Does any of this matter? And so we had a conviction from early on that that was going to be the kind of crux was to pass this you must get tested every quarter basically run thousands of simulations to see well so can it actually be jailbroken? How hard is it to jailbreak?
17:26How often does it hallucinate? How often does it leak data etc. And then the last uh core idea here if you scroll up to the top here is to refresh it quarterly.
17:37So the core trait of AI is that it moves extremely fast. whatever concerns we're discussing today were not the same ones 3 months ago and this will keep changing typically standards update on like a decade cycle is obviously not going to work but the question is kind of how do you update it and the core thing here was to basically get the risk leaders of the fortune 1000 around the table so if you go over to the left here you'll see as1 consortium the consortium is a group
18:04of risk leaders who run real banks real hospitals real critical infrastructure who are facing these challenges every day and we meet with these folks twice a quarter and hear what's top of mind, what is keeping them up at night.
18:17There's tremendous amount of desire for that conversation and then we operationalize that into a specific standard that gets into and actually we can go into and look at what is what even is a standard. If we go back to introduction out there to the left and scroll up a little bit to the wheel click into reliability. So if we take something like hallucinations hallucination system reliability there is a number of requirements here. If you go into the top one, prevent hallucinate requirements uh hallucin outputs. This is one particular requirement. This is a technical control. Basically, we want some kind of groundness filter. The
18:46first thing you see here is what's called a crosswalk. So, everyone and their grandmother has put out a framework, very high level framework for what are the air risks.
18:56This is basically your competition. But in some ways, our competition, we're in fact friends with them. We'll come back to why. But mapping everything together. So, you have one superset. the the claim you're trying to support here is uh if you follow this framework then you can also see how you follow the other frameworks but the meat of it comes down here in control activities and evidence.
19:15So control activities is like great you have this high level requirement how do you turn that down to something operational here's what you must do and then what is the evidence that we're looking for and the reason we go this deep is that there's actually not that much confusion about what are the big concerns in AI everyone agrees to these the question like what are you actually supposed to do and so what we found a lot of demand for is getting down to the specific evidence uh that people need to
19:43look for whether you are cursor building something or uh even J Morgan building something but also if you're just a risk leader at JP Morgan like what exactly should you ask for what can you ask for without sounding stupid like if you ask for something you won't believe the amount of time a risk leader has asked for the IP rights to the online model to cursor or something and you're just like sorry what like you slip it in there and see see if you notice see exactly put that in the
20:11questionnaire so that's kind of what a standard is And we update this every quarter with these folks uh to keep up with the latest consents.
20:20Can I double click on this one?
20:21So, first of all, the website's beautiful. Like, it's so confidence inducing, which is the whole point where like like okay, I know exactly what I'm signing up for when I when I talk with you. I don't even have to talk to you. I can just see your whole uh certification, which is great. But like, okay, so from from here like D001.1 config ground filter filter, how does that get applied? like you have a person yes goes through it. If you uh go back I can see somewhere there's like you
20:50know 51 requirements 130 controls there's like a whole right I just like to me this doesn't translate into a test or yes yes yes so if you go into uh out on the left hand side so actually if before we go in there there are three types of requirements the first is technical controls like you must implement some guardrails two there are test controls so you must have an independent third party go run some tests against you. Well, I'll show you one of those in a second. And then
21:18three, there are policy controls. For example, you must have a person whose name is on the line when you guys up.
21:25And you must have a plan for how you tell your customers and how you engage with them. They're kind of more traditional standard type stuff.
21:32So in this particular instance, we just check whether they in fact have a ground in this filter. So we will partner with an auditor. So we partner with auditors like KPMG or like Shellman who go in and do the thing auditors do, which is to check the evidence. In this case, that might be a screenshot. It might be part of the code that they need to review to see that it actually just that it exists.
21:51And then the second thing, so you're not testing the effectiveness of it.
21:53That's the second thing. So if you go down to the third party testing for hallucinations out on the left, that's basically the next requirement. This is where we test how well does it actually work.
22:01Okay. And is it you testing or the auditor?
22:05That's a lot of work.
22:06How long does testing take? So if I want to get certified, just how long does the end to end roughly take? Yeah, the end to end uh almost always is dependent on the our customers need to learn something for us. It takes somewhere between like 3 to 10 weeks depending on how up to snuff they already are. So some people show up to us with like extremely rigorous security programs. When we test them, it works extremely well. We can get that done very quick. Some people come to us and they're not that far along. We give them kind of the spec that they need to build
22:35towards and then their security teams and engineers get to work and build to meet the standard. Uh the testing itself typically takes a couple weeks including the time for them to remediate. Often we'll find something that we cannot pass where hey this is actually just not up to the standard. You won't pass the standard and then they will need to go and implement additional safeguards or additional remediation that makes them more robust so that they can actually kind of hand on heart look at their customers in the eyes and say like hey we've done truly our very best
23:04and they're certified for a year and have quarterly updates.
23:07Correct. Yeah. And yeah, it's pretty interesting. I think uh you know what's changed since? So this is certifying agents in production, right? Your customers like you've had lovable, 11 labs, intercom, they've all gone through this certification. Uh what has changed? So I see you post like you know Q2 added MCP agent um agent agent communication. Any other things that you want to kind of highlight since the first first iteration? What comes in quarterly?
23:32Yeah, so some of the changes have just been ages are not just one thing. So like if you take agents like cursor and compare them to Sierra, they're really quite different. And compare them to Harvey again, compare them to you again, 11 Labs, they're all quite different. And so we wanted to design a standard that works for all of the types of agents.
23:54And we started with one that was like pretty textbased, like honestly pretty customer support focused. That's where there's a lot of existing demand. and then over time have picked uh some of the frontier companies in each of these other domains that we could work with and build out the standard. So such that we know that the same standard works for code, it works for custom support, works for automation etc. So that's been one big thing. Yeah. Then some of the things that have been top of mind recently uh Mythos is bringing up a lot of concerns for security leaders. We're starting to
24:24get more and more questions around agentto agent interactions. It's very nent uh at the moment but is starting to emerge. There've been a lot of uh questions related to open claw and MCP again like agents starting to interact with each other uh is really top of mind then as coding agents have really taken off. That's also where banks and hospitals etc are getting more and more precise on what it is they need. So really dialing in as as to be like where most of the tokens flow through in the
24:53world getting much sharper on that. Can you share for people that are listening that don't really think about this? Like you mentioned there's the obvious stuff, you know, hallucination, citations, what are best practices that people should do when building agents like if they come to you pretty ready with certific like, you know, they'll probably pass certification. What are the things people don't think about that they should have? The most important thing is that a lot of companies have not done a serious stress test. They spend most of
25:22their time, perhaps rightly so, optimizing for how does it work in the good case, the average case, how high quality is the output for the customer.
25:32And a lot of these companies are pretty new, so they haven't spent a lot of time stress testing the what is what is there as an adversary on the other side. What are some of the complicated corner cases that you've not really considered? So, I think that's like a frame of mind and you'll also see this in startups. It often takes a while until they hire hire their first security person and that's a whole different kind of risk surface than just building a good product. So a lot of that applies. Most companies actually also have the right kind of architecture. Most of them will have some kind of guard rails in place.
26:02Either some come out of the box from their model provider or they'll have built their own filters to sit in between. They just don't work very well.
26:08the difference between putting a classifier in place that like maybe goes and checks whether you're giving medical advice when you shouldn't and says, "Hey, if this looks like medical advice, filter it out." Lots of companies have that in place. The question is whether it works. And it's actually pretty fiddly to sit down and think about all the ways in which you could ask for medical advice. Read the academic literature and what are the kinds of framings or tricks you might play to get an AI to give you medical advice when
26:36you really shouldn't. Uh and so there's like an area of expertise that's just missing. So what we find is that most people have the right building blocks in place that it doesn't it's not rocket science but the finicky thing is like getting into the corners and testing whether it works such that you can look your customers in the eye who may be a bank or maybe a hospital and be like this is this is going to work for you. I see. So we talked a lot about the agent level certification. Where do you guys go from here? So announcing series A off
27:05camera we talked about this a bit. Um there's the whole security risk of fable government stepping in. You guys are kind of announcing that you're also going into model certification.
27:15When we do a bit of cutting afterwards, we we will not yet be announcing this.
27:19The question that is top of everyone's minds now is at the model level and Mthos then Fable has really brought this to the four that in addition to the commercial risk and the kind of economic security risks that are happening at the agent layer the models are going to present risks in the national security category. The shape of the problem is very similar. You have some people that are on the hook if something goes wrong.
27:44In the case of agents is often the security leaders in the enterprise. In this case, it's the government. They don't haven't necessarily spent their entire lives thinking about what are the new risks that come here, what is the kind of data you might be looking for, how might you test that? But they do have to make sure that their concerns are addressed. You have some frontier companies that are deeply technical.
28:04They know a lot about the risks, but they fundamentally have an incentive to not always be truthful. So, you have a trust gap between the government and the labs. And in every other industry, you end up with some kind of body sitting between a neutral third party sitting between those people. There's no other industry where we allow people to audit themselves. So there's going to be a need for a third party that can take the rigor of the labs to run frontier
28:31technical evals, but can also speak legible trust in the way that the government trusts PWC to go and run financial audits. and they know that they output order reports in a way that's consistent, that's easy to read, that's factual, that's uh trustworthy.
28:47Those two things need to be brought together. And what we've learned from our work with agents is that if you want those that communication between those two parties to be smooth, there has to be one common standard that that is public that people can go and inspect.
29:02What are the risks that matter within each of these risks? What are the kind of threat models that you're really looking for? You need to specify for each of those risks, what are the guardrails that need to be in place and what are the tests that you need to run to see whether those guardrails are effective. And then you need to go and run audits that are technical audits that are consistent. So if you're trying to bring trust, it's extremely important that you methodically work your way through the risks. You can't send one researcher in and say like come back
29:30with whatever you find. You need to be able to explain exactly what you did, exactly what you tried, exactly what you did not try. and therefore the kinds of promises you can and cannot make at the end of it. I think of fable as a direct symptom of this problem that the government was told that there's a risk.
29:48The government may struggle to assess just how big that risk is. They call anthropic and anthropic is trying to tell them, hey, actually every model can be jailbroken.
29:59That's not what you want to hear, right?
30:01as a government that might be hard to trust and we think that a broker is the most natural solution. In other markets you you see something like uh in financial markets you see Moody's Moody's goes in and they look at a bond and they output a rating. They say like here's the evidence we found. Here's the rating. We don't decide whether anyone should buy this bond or not buy this bond. Well, that depends on their risk appetite. But we do proide this common information
30:28layer that everyone can rely on. In the case of Moody's, the government uh points to them and say, "Hey, pension funds, you should probably really take care. You shouldn't risk your pensioners money. So, you can only invest in AAA rated bonds." That means that now the government doesn't have to staff thousands of financial technical experts to rerun forecasts every week to see whether things are correctly rated. They get to point to some neutral third
30:57party. Uh so my hypothesis is my hunch is that you will see a third party that sits between the government and the labs and it could either be the government builds it themselves. So something like Casey was set up to do exactly this. And the question is, sorry, I'm not familiar with KC.
31:14KC is the center for AI standards and innovation.
31:19I won't get into the details, but it's a sub body of NIST that typically set standards. Uh so it's basically a government body that has experts.
31:26Yeah. Exactly. Very very low key.
31:29I think you know it's one of those things where when you just sit back and listen look at it like is there enough technical expertise in the government to measure test these things right now?
31:40Probably not, right? And Fable is a result of okay, we've had to scale back and pause things, but and they have they have excellent people uh but they have an extraordinarily small budget compared to the scale of the challenge that's ahead of us and I think they have a role to play. The question is kind of like who does what and we have now outlined the jobs to be done and they're quite extensive. Every model release there is an astounding given that they they take in any input
32:09the risk surface is astounding and so the question is really what can only the government do and what can the market provide here that can keep up with the pace as AI risk changes. Our perspective is that also at the model layer the risk that people care about today are not the same ones they care about 3 months ago. So the pace of legislation is too slow to deal with pinpointing the risks here.
32:31And so we think there's a lot that the market can do to surface timely information. Ultimately there's a bunch of policy decisions here. Is the national security risks of a model too high as a political answer.
32:42Uh but what you want to make sure is that the process that produces this risk information is compatible with very fast innovation.
32:49So you don't want to this is not a question of like can you slow the things down? Can you keep the models locked up until for months on end until everyone can make a guarantee. uh but it is those can you in the time it given that the US is competing with China on releasing models can you insert risk information that allows the government to like make rapid decisions on some of these questions balancing that trade-off between failing to adopt AI is going to put us at risk but also reckless adoption it's going to put us at risk
33:18and that's a very kind of fine balance that they're going to need like a lot of high quality intelligence to to make just a side mention because you mentioned Chinese models any specific big concerns that you're hearing from your CESOS about that uh cuz I guess it's free but CISOs have a bunch of concerns around data flows in general that they're really concerned about. So there's a lot of questions like if these models are Chinese where does where does our data go? I think a lot of this can be addressed but they they come up often.
33:47I mean they understand they're running on American GPUs. Some of them some of them understand cuz they're running on American GPU.
33:52They're not like phoning home every time you like call home.
33:55No. uh a year ago there was not a lot of understanding of this. I actually think uh you're seeing the security leaders becoming kind of AI literate at a blistering pace and you're actually also seeing my Twitter timeline that's very AI pilled and my LinkedIn feed that used to not at all be AI kind of converge. They're both talking about fable, right? Yeah, that's true.
34:15They are both talking about whether you can prevent models from being jailbroken jailbroken these days. Uh like national security risks are kind of that that conversation is actually emerging. Other than that, I think you mostly see a kind of uh there's no concerns with any particular model or any particular model output, but there's a general nervousness of having critical infrastructure run on models that are not produced in America by Americans where the American government has control.
34:44It doesn't necessarily show up in your framework that directly or it might there's a bit of stuff in there actually on the like the provenence of the models and disclosing that. But I think there's a bunch of use cases we're running and Chinese open source model is just the best solution. Uh and a concern is slightly more macro here which is not best addressed at any particular certification level.
35:01Is there anything interesting that you see at the you know if you're trying to fill that middle gap that mediation gap any interesting stuff that you guys forecast would be required other than you know what what the average person might expect?
35:16There's a bunch of interesting questions about what are the risks that matter here. So right now the risk of the day is cyber because it's very real, very tangible. Um some of the risks that are also emerging as pretty real and pretty tangible are things like child safety is becoming both extremely important uh but also politically important. And there are some of the risks that are coming down the pipeline that today feel kind of speculative, but people who spend a lot of time with the models see them coming down is things like um risk that
35:45relate to biology and specifically where the models will help adversaries produce biological weapons and making that extremely cheap, extremely accessible, producing making the chance of another CO or worse pandemic. CO was not engineered to be bad. uh as if you were trying to do that. So I think those are some of the risks that are coming down the pipeline. Uh I think one other thing to just note is that agents are kind of
36:12deliberately narrow. So like when a frontier agent company puts a chatbot that interacts with customers, they've really tried to narrow the topics is interested in talking about such if you ask it like what do you think of the president? It will just decline which means that the kind of risk area is so much smaller. For models it is infinite.
36:32And so there's not a single expert out there who can competently evaluate the risks of cyber attacks and 15year-olds having month-long conversations with a chatbot and seeing whether it will in fact recommend suicide or something horrendous like that and can evaluate the risks that terrorists can use AI to produce boweapons. The risk surface is just too big. And so the central challenge actually becomes how do you
36:59get those subject matter experts to work within a one coherent framework that outputs one coherent report and rating that the world can go and inspect because that global perspective is central but there's not a single organization today that could produce that and you would be the presumptive one when you put out your model standards.
37:20We think there can be one company that can with a consortium of experts build one coherent standard. I think we've shown that across all of the enterprise risks today. We think there could be one company that could with a consortium specify the audit rules basically like the inputs and outputs that all these technical experts need. What access do they need? How should they treat infra infra security? They can look at whether the eval evals are well produced without necessarily being able to say hey is
37:49this a thread or not a thread but overall evaluating whether the evals are good well constructed that set of older rules that basically becomes the interface for all these experts we think one clearing house could put together to be clear when I say one company I think of it as one company coordinating lots of this in the same way that when we saw our consortium it's not like we say we have all the answers on agent security.
38:13What we say is we are taking on the role of eliciting all of the concerns and being the secretary that puts it together and runs a tight house such that the standard updates lock step every quarter and that the order reports that come out in this case 100page order reports uniform and crisp and clear all to the level of detail is required for executives that need to make a clear go no-go decision. So that's kind of the role that we think we might play. I think in many ways you're performing the
38:42role that OASP used to do there and you said like you know competition and partners. Can you go more into like how they partner?
38:49Yeah. So first of all OASP is basically an open source community of security practitioners that are coming together to build frameworks for addressing the latest security concerns. We think they are phenomenal at creating frameworks.
39:01We've in fact we've first of all we're partners with them. So we have a joint article two. We've learned a lot from them. we think a tremendous source of of intelligence. What does not do is building the machine that runs third party audits such that a company like Cursor or a company like JP Morgan could get a third party to go and review them against this and say hey you've passed the standard and here is the report that you can use to build trust and preempt your partners or customers questions. So they
39:31fundamentally try to do something different. You can they are part of the information gathering and intelligence gathering and creating clarity but the operational layer of turning this into promises is is not the business they they tried to be in.
39:43The standard is emerging and and it's doing very well. Was it necessary to then also do underwriting? Uh obviously it's in the name so presumably you thought about it first. I feel like if you just have enough consensus you don't actually need the money angle but it does help. I I did want to also note you guys are a for-profit company too, right? It's not nonprofit work. There's there's a whole business side to it as well.
40:08Yeah. Yeah. Yeah. I'm crazy about the money.
40:11Yeah. Yeah. Yeah. Let's get into the money part.
40:13Let's start from actually your question.
40:16Forprofit versus nonprofit in the security space today. Cyber security most of the standards are produced by nonprofits. I think that's an issue.
40:28The question you have to ask yourself is how do you create good incentives for these standards to be good and keep up?
40:37Nonprofits tend to not have these adverse profit incentives where they uh hollow out their standard and create a race to the bottom, but they're also not at all responsive by default to the communities that they serve there because there's no process. They don't have customers that they serve where they go and ask what do you want? What do you want? What do you want? And when you look at the overall satisfaction with the security standards today, people tend to just not like them very
41:04much. You do see in other domains uh that for-profit standards can serve the world quite well. So there are examples uh like we talked about Moody's before.
41:16It's not without flaws, but uh it is absolutely critical societal infrastructure that gets run at astounding scale today. your credit score. It's FICO. It's also a for-profit business. And when you go back even further in history, some of the crash testing standards came out of insurance companies. The insurance companies together funded the founded the institute of insurance institute of highway safety because they were very interested in like how can we use
41:44standards to drive down mortality and save money.
41:47Go back uh our name actually pays homage to the underwriters laboratory. UL which uh was started right around when electricity came out. Houses started burning down. Insurers again were paying the bill and they were maybe also good people but their profit incentive was let's prevent houses from burning down.
42:06Let's test all the electrical products the light bulbs all the light bulbs in here are probably UL tested the toasters etc. And they set up uh an entity to create those standards. Today, UL has a for-profit entity and a nonprofit entity. Uh, what they've recognized, they spun out, they start a nonprofit, they spun out a for-profit because what they recognized was like, hey, actually, to serve customers well, you need a for-profit entity. The lesson here is one of the ways that the market can align incentives, so you're both
42:34responsive to customers and not hollowing out your standard over time, is to align it with insurers because they fundamentally have good incentives. And so if you're a for-profit standard that works closely with insurers, you get the feedback loop in such that you're really curing to your customers but also have their interest at heart. So that's the model that we're the kind of inspirational model that we've learned a lot from and that's also where the name comes from.
43:01In some ways the the term underwriting can both be associated with insurance, but it's also a broad term for like making decisions. M if you underwrite a decision uh you're fundamentally kind of taking ownership for for the consequences of it.
43:14Yeah. I mean what does an insurance contract look like for AI?
43:18Yeah. Most of the demand comes today for insurance contracts is uh sitting between people who've built AI and people who are buying AI.
43:26And what you want is the reason why people want insurers involved both for the traditional reasons. Hey, if something goes wrong, we want to be compensated. But it's in particular because insurers can increase can bring trust to the equation because insurers will take pay for the damages if they're willing to write an insurance policy.
43:46That is them saying hey we think there's risk here but that is manageable and that is kind of a their incentive aligned with the enterprises adopting it. So that's a really a good signal to the market. In the same way actually uh one of the things that Whimo tried to get their first permit to even operate in San Francisco was to get a lot of insurers to stack up a huge insurance policy in the case of something went wrong. Not because Google can't pay, but because it was very valuable to have a third party go and look at that data
44:15that are trusted by government, trusted by enterprises as conservative people and say, "Hey, we've looked at it. We're actually willing to take some of this onto our balance sheet." So that's that's kind of the reason why people are interested in it. What it looks like is uh in some ways like every other insurance contract you specify what are the perils you want to cover. How much do you want to cover them? Like up to what limits and what does it cost to cover that? And in the case of um if we take a really concrete example uh 11
44:44Labs uh bought a first of its kind AI agent insurance policy. They work with some of the biggest uh enterprises that work with governments. They're really interested in going above and beyond and making promises to their customers. So, they wrote a policy that covers just some of the core concerns that our customers have been asking about. And uh the crucial thing was really to get Lloyds of London, the world's oldest insurer, one of our partners, to look at
45:12this data and be that third party alongside us to say, "Hey, we think there's something here that's worth underwriting." Um, and that's actually what it looks like. And so they will show that c that contract to their customers and they can see how much they're covered for. They can see what exactly it covers. Uh, and that will also probably change next year. They will want to write an insurance policy that might cover more.
45:33When you say Lloyds, is it reinsurance or are they sharing somehow at the same level or Yeah. So typically the way uh new companies get into insurance is that they partner with insurers such that the insurers take the majority or all of the financial risks. Fundamentally if if if insurance is useful because it brings trust you have to be able to pay the bill. Lloyds of London is 400 years old.
45:59They've never not paid a claim. They're extremely trusted. Um what Lloyds of London struggle to do on their own is to figure out which of the risks are real.
46:09what should we be looking for? What are the kind of technical controls and running the tests? So they use AEC1 as kind of the underwriting framework and we produce a bunch of e results that then directly feed in to inform the pricing. Uh so this means that 11 Labs customers know that that payment will be there. They don't have to look to our series A and see like do we think they have enough cash on the balance sheet?
46:32They will look at Lloyds.
46:33Yeah. Um and Lloyd's like famously very creative. I I think I remember some headline like they insured Jennifer Lopez's butt or something.
46:42Correct. And I think uh was it David Beckham's right foot? Yeah. And stuff like this.
46:47So like clearly not a large data set.
46:53It's actually a remarkable institution that's both kind of has some of the truly old school virtues of having been around for a long time. they like really they really operate like a trusted entity and they have appetite to figure out the future h and I think there's a lot of recognition that both there's like tremendous amount of risk in AI that is poorly understood today so getting into this business carries real risks uh but also this is where lots of the risk exposure will happen in the
47:22future this is the one market where risk is truly growing this is the one market that will also take out some of the existing thing Mark is take like auto insurance when there are no human drivers how's that market going to look well it's clearly going to change how are you going to assess you want to ensure way more I all I'll say is the principles for how you insure way more are very similar to how you ensure other kinds of AI so again crash testing that's what we do for customer share are lovable that also need to happen for Whimo which is not
47:51how you do it for human drivers so there's this growing awareness that the world is changing very fast and the only way to learn how to underwrite AI is to write some policies. You may incur some losses and and think of that as R&D expense really. But the question for them is like who are the trusted technical partners they can get into this business with that can help them navigate and make sure they don't make uh kind of foolish mistakes. But also who is willing to hear the wisdom that they have? They've done this before.
48:18They've seen they were there when cyber came out. So there are lots of ways in which AI feels completely new. But there's also lots of ways in which risks look the same. And so there's actually tremendous amount of wisdom sitting in some folks that may have gray hair uh but really have like a a keen sense of uh how to quantify risk.
48:37Yeah. And and the number is so it's basically like I want $50 million worth of cover of coverage against these perils and voids will give you a quote on it and then you you have like a small markup or something and then you you turn it around and and do that. Is that is that as simple as it is?
48:52You you basically share some of that premium. X% goes to the people who do the the pricing of it.
48:57It's kind of like a it's kind of like a merchant bank for insurance type of thing.
49:02Exactly. You basically split the fee and you can think of the insurance supply chain as like there's bringing the capital, there's doing the pricing and there's doing the distribution and typically you will pay out some x% of premium here, y% of premium here and the rest of it will go here. Does all the insurance world work like this or is there some point at which like so so right now you have equity capital at some point maybe you start raising uh debt or whatever and then you have enough of a bank account and enough history let's say you've been operation for 10 years that you don't need lawyers anymore
49:30that's totally an option uh and I could see some worlds where that makes sense specifically if there are risks that we feel high confidence that we'd want to insure where the incumbent insurers are too slow to find appetite okay or or simply struggle to evaluate s they don't want to do But by and large in general you do not want to compete with insurers on uh bringing risk capital to the game for two reasons. One is that's fundamentally a cost of capital game. They have extremely low cost of capital. Startups have high cost of capital by and large.
49:59And two you want to hedge your bets and it's very helpful then to also have a portfolio of home insurance of car insurance. And we we're not about to become a car insurer nor a home insurer. So they have some natural advantages which makes it much more likely that we'll partner.
50:17And they bring that the capital at scale and we bring the technical.
50:20You're going to work with them for a long time.
50:22How are the discussions with the insurers as well? So basically they're going off of your certification, right?
50:27They're trusting the diligence on you that your certification is valid. You tested the right things and they're backing the money that you know you have the right testing in place. So any interesting takeaways from working with insurers? I think the movie the first thing is they feed into the standard as well. So if there are things that they feel like they need that they're not seeing we are also taking that as input into the standard uh because fundamentally we think a good standard is one that creates a really healthy
50:55promise ecosystem and we think insurers are critical part of that. Uh and again they are the most well incentivized to they see all the loss data across any particular CISO knows their particular concerns. insurers see the concerns across the entire portfolio and often have direct access to like what exactly happened, who was at fault, etc. as they do part of their forensics. So, they're actually like a great source of intelligence on this. One of the big takeaways from cyber insurance, which is a market that didn't work that well, was
51:23that the insurance and the technical expertise was not married up. Uh what our conviction is that standards have to precede insurance. fundamentally what everyone first and foremost want whether you're a CISO at Jig Morgan or a CISO at Cursor or a underwriter at a Lloyd of London syndicate is you want to not have an incident in the first place. You want to know that the risk is well managed and only then does insurance start to
51:53make sense. So we'll see the standard ecosystem basically run ahead of the insurance. And the reason why we you asked us kind of why I also do insurance this is kind of proving what we think that whole promise confidence infrastructure ecosystem needs to look like and we think it's very compelling to bring that to life even if we think the standard is kind of the the core lynch pin that unlocks the rest. There's been no claims yet right?
52:15This is one of those things where um you know if people haven't really worked through what it means to cover things. So, for example, I pay cursor $20 a month and I write I vibe code something that makes uh a plane crash causing $200 million worth of damage.
52:31Uh, do I claim $20 or do I claim 200 million?
52:36Yeah. So, and these are all great questions. Uh, and fortunately kind of all of insurance and legal history kind of helps answer some of those questions.
52:46I think the first thing is people have limits on their policy. So if you want to claim $200 million, you have to someone has to have paid a lot for that insurance policy up front to have $200 million of coverage. And ultimately the way this works is that uh you start from a lot of uncertainty. This is not just an insurance but also like can you use cananthropic use books from the internet to train up well they can go and look at precedent they can see but ultimately this these things get settled in court and you hammer it out
53:15over time. So you start from this like place of ambiguity which is both why insurance can be hard to do early on but it's also why people want insurance because that ambiguity slows down adoption that also sits at the heads of the uh in some ways actually the first incident will help to like establish a lot of this exactly and and there have been a number of incidents out there that have just not been insurance covered. So take the now old uh example from Air Canada where
53:44hallucinated a a refund policy and the question was Air Canada in that case were like hey we have nothing to do with this chatbot messed up but like sorry and the courts were like no if you put your chatbots to interact with your customers they make legally binding promises on your behalf. That is now precedent for everything in the future where you will if someone were to deploy a chatbot like that again they should not expect to be able to just pawn off
54:12and say sorry my chatbot lied it's nothing to do with me I bought it from open AI no if you're putting this in front of your customers you are taking responsibility for it and so every court case whether insurance is involved or not clarifies liability and liability is kind of the foundation for insurance there's another reason why stands and insurance come together liability for I'll go on a little tangent here. Get into the weeds of it.
54:36Liability often one of the core concept is whether someone was negligent. Should they have seen this? Should they have prevented this? And the question you how do you judge that? Well, you basically judge whether they've met their duty of care. What does that mean in practice?
54:51Well, often they looked at standards. So if there's a standard that is broadly adopted that says you must have a groundedness filter or you must have a jailberg filter it becomes way harder to claim ignorance that these things existed and so setting standards help clarify liability points courts will often point to standards and being like well this seems like best practice to do is there for everyone to see. So there's another way in which like standards are kind of civilization infrastructure that insurance can build on which promises
55:21can then build on. I I totally get that we don't have to get certified to to write these to you know make these like bots and all these.
55:29Um but like basically whenever we get go for the audit I think people like start to shape up and and all this all this stuff. I wonder if like that means that you don't also then become like the approving authority for me to ship to production you know like um yes you check once per quarter I want to ship once a day.
55:48Yeah. and I don't know when one of my things breaks like one of your certifications or not.
55:53So there there's a couple of things um there's a couple of requirements in there that relate to how do you yourself where you have to tell your customer how are you yourself testing before you make at least major releases. We don't go and order to people every day. Uh but at least there is now a trail where if you do a major mess up then your customer may come and ask you hey you promised me that you were going to run these emails yourself and for lots of them most most of the PRs that people merge will not
56:22fundamentally alter the product experience but some of them will and sometimes you don't know and this is also true and this is also there's some inherent risk that everyone kind of everyone knows that when they buy software there can be bugs and this this is just part of it but what they can if you're selling to mom and pop shops they may not care that is like well I want to use your tool so I'm just going to will be willing to take that risk on if you're selling to a big bank they might be like sorry we're making promises to our customers if you can't make a promise to us that we can pass on we
56:52don't want to work with you then it's up to you to say do I care for my agent to get used as critical infrastructure in this nation if so at least I can make promises about what process I run and then we can go and test it every quarter to be like well does it seem like uh it's if uh kind of it still meets the standard. So from my perspective, it's kind of a way to big companies by default kind of have some amount of trust when they ship AI. If you're a young company, if you're just starting out, by default you have no trust. And
57:21there are very few places where you can go and get trust. So one of the things that most of our customers did before they started working with us is that they would make their own security blog posts. That's great, but also who's going to trust you saying we're so secure? Like anyone can write that, but it's very hard. Where do you go and get that trust? Yeah.
57:37And so I think making the standards more legible makes it easier for smaller companies to prove that they're doing what they ought to be doing because the default assumption is that it's the wild west.
57:49Is there a road map you have of like there's a lot of work to be done here, right? This is the first one. Um anything on the road map of what you see is next, what's coming, what's what's missing? I think when we when we zoom out a1 deals with agents we will next up we will deal with models next up from that we will deal with robotics of which in some ways Whimo is the first robot but the exact same problem is going to be someone's going to develop a robot someone's going to need some promises
58:19they're going to struggle to make the promises and you see this playing out when like uh if you think fable concerns are bad like see when hits a dog and as if people lose their mind imagine when first robot knocks off a toddler off a kitchen table.
58:32You're going to see some real strict liability.
58:36I mean, you can see it, right? Like crews got fully destroyed.
58:39All permits are gone. Yeah.
58:41So, physical AI, the level of stringency just goes up and up and up and up. Um, so that's kind of like the big picture.
58:49Agents, models, robotics. I think within agents the current set of agents are well covered by this but as the technology progresses as agents get longer horizons new types of failure modes will emerge and so it's mostly of can you make sure that the center keeps up when they appear and you also start to see new modalities like today world models is mostly kind of a a research question there's no one who's really using it but that will also bring in
59:18just new kinds of ways to create value but also more risk surface that no one knows how to grab it with today. You'll start to see true agent to agent interactions that are not mediated by humans. There's going to be a bunch of interesting questions. You're basically going to need a new legal system. How do they build trust amongst each other? How one of the core things when humans trade with each other is that you know that you have recourse, you can sue them. How do you make sure that there is a persistent balance sheet behind any agent such that if you trade with it and it screws you, you know you can get your
59:47money back? Those are some of the questions we're going to have to deal with. And the technical testing of multi- aent systems is also going to be interesting and complex.
59:58Very fun. Uh are there any perils that are uninsurable right now that people wish that you would?
1:00:04Yeah, one of the places where there's a bunch of appetite for insurance and not a lot of a lot of demand but not a lot of supply is when it comes to copyright. In some ways, copyright is kind of mundane. It's always been an issue. Uh there's a couple of reasons for this. The first is people who have trained on copyrighted materials almost always know that they've done that.
1:00:28So if you want to buy insurance for it, it probably signals that you might be a high-risisk customer.
1:00:35The people who are most interested in getting insurance for copyright infringement are the people who are most likely to have like it's like a lemon problem.
1:00:42I actually think there's another side to it too, right? Like if you're building on something, so say I'm using an open model, I don't know what it's trained on, right? And how far down that chain does copyright go? Yes.
1:00:53Am I liable to take down my product because company X train?
1:00:58But there's safety in numbers. If everyone's doing it, then you I mean I would say until, you know, Fable is rolled back from everyone that use it, right?
1:01:06Yeah. This a hard question. I don't have the answer to that but I think your your intuition is your intuition is right that kind of like uh what is the kind of duty of care and people don't today think of it as customary that you go and you like dissect your open models training data and you check everything. In fact lots of people use them. It's seen as kind of generally acceptable to not check for this and therefore we're not going to hold you specific.
1:01:31We also really can't right we don't exactly we don't know the training data. You can ban it, but I think no court is going to get a copyright question. He's actually to get banned.
1:01:38Hire Nicholas Khini and he can extract it from Exactly. Though he's in short supply.
1:01:44Yeah, he only has so many Carolines. But uh Exactly. So I think this is this is also fair that uh in the case of labs there's a lot of interest for this, but the thing that makes lab wanted is what makes insurers suspicious of it. And so you have a lemons problem. Um yeah. Is there like a theory of insurance where adverse selection dominates the risk sharing aspect of insurance? Like where does this like teach us insurance?
1:02:09A lot of insurance does come back to like practical versions of microeconomics 101.
1:02:15It's like it's like this is why you need to pull health insurance uh because if you make it too hyper specific then only people who are guaranteed to get the disease will sign off for your insurance.
1:02:25Exactly. Same same thing.
1:02:26The core problem is one of information asymmetry. people buying insurance know something about their risk that the insurers do not know. And so the question is actually and this comes back to the same problem is if you rely you can break a lot of these information symmetries if there is some kind of testing that reveals the underlying true risk. And so if you were able to in the case you mentioned have good diagnosis of whether someone has it or what the probability is that someone has it that the insurers trust then they might be
1:02:55willing to insure it. But if they don't, if there's no kind of common information, then they only the patient will know. That's what breaks it down.
1:03:02So the question is again, how do you create credible signaling between players? This is also the whole reason why Moody's exists. Moody's just does credible signaling. That's also why Moody's could never uh Moody's has to be independent. If Moody's was owned by JP Morgan, then JP Morgan could not use it as a signaling mechanism. So a lot of the basics of standards and certification are just communication devices. there's just a trust gap and uh that's where you have to think about
1:03:30what are the incentives of the messenger and one another way you can break a lot of this is through transparency. If you are transparent in how you operate you just cannot mess with others nearly as easily you make it much more costly and that increases trust. This is one of the reasons why there's a change log here.
1:03:45Every little change Yeah. Yeah. you can go back and find and it means that if we were to make the standard worse Oh wow, that's a lot of changes in one update.
1:03:57And a lot of this is just as things get clearer. You can see a lot of clarifications. You can see some revisions. As things get hammered out, you want to change this. But if you make it all public, you make it much harder to mess with people or at least you become found out very easily. Mhm. And so this is a way of increasing uh so reducing the information as symmetry is by just making more of the information public.
1:04:18I like how you do know when future versions are coming. So I guess it's just not that surprising.
1:04:27Yeah. But this is also a promise like if we now don't deliver on July 15th I mean you can just batch it up and then whatever you got. Yeah.
1:04:36Like we deposit some amount of trust every time we meet this commitment.
1:04:41Uh and in the startup land it feels easy to ship a new version of a standard once a quarter. Uh in the enterprises who are used to this like decade long cycle we often get met with like incredul like there's just no way and then you show them the change log.
1:04:57One thing I wanted to also like try to really think about is you know you said something about how if you have tests for the thing then you can ensure it.
1:05:05Right. And so really what your standard is, what AI is is establishing a framework for the audits to happen so that you can at least test like all these like baseline standards of care have been met and therefore people can ensure against standard risks that everyone has. I wonder if like there needs to be develop you need to develop other tests. Uh we've covered mechan uh in the past. Any interest in that or are there other kinds of tests that we're not thinking about? Yeah, I think
1:05:32Mechinto is a big one. Uh, a lot of interest in that. I think everyone would agree that there's like promising scientific potential. We're still a while a little bit away at least from this being like commercially available on demand such that there's like now a selection of vendors you can go to.
1:05:55Goodfire would say it is commercially available.
1:05:59We would agree with them. We we think the work that they're doing is tremendous. We're not quite at a point where we could like literally require it, but it's the kind of thing where you can imagine relatively soon you could put in an optional control for if people use that interpret as a way to reduce risk. You at least get credit for it. We can't require it because it would be hard to require everyone to become good fire customers.
1:06:18What good does credit do me? This is a past fail, right? Do I do I care about credit? It's a past fail, but it's also 100page order report that you'd be surprised at how much the current leaders actually sit down and digest this stuff.
1:06:31Uh and I promise you that if someone is using mechan today, uh they will have a slide on it.
1:06:40It is cool. It's fancy. Yeah.
1:06:41But it's just easier if you have a third party saying, "Yep, they have mechanics."
1:06:45just to like just to flesh spell it out for people people who have been following our our mentally like oh you're using uh you know GB2SS it is activating these three dangerous things we monitor for it and we log it out in whatever tool of choice grawine has like signal whatever and that's it that that that's the mechan based activation uh signal okay yeah yeah so I think mechan is interesting and I think if that promise truly comes to fruition you can make stronger promises than you can with
1:07:15evals. And so I think that's very compelling. Another thing that I think will become increasingly important is just kind of good old school monitoring and slightly after the fact. Uh one of the things you're seeing with Eval, some of the challenges that are emerging is that the agents are starting to become aware that they're being eval.
1:07:34Exactly. Which is a problem. It means that they basically if they know they're being watched, they won't do the thing that they think they get punished for.
1:07:43And by default, unless you know how to kind of reduce your awareness, you should trust El less. And one of the kind of truest things monitoring like is the source of truth. Did you in fact give medical advice and how quickly do you know? How often do have you done that in the past? How fast do you respond? How often do you detect it? How fast do you detect this? Uh so I think that is also a paradigm that is slightly more intrusive. You actually will look at some customer data. uh but I think
1:08:11will become more prevalent over time.
1:08:13People talk about this like we should not write about EVA awareness because it's going to leak into the data set and then beat like we should just like we should like never talk about it only meet in person and like talk offline unrecorded like did you guys see the anthropic research where I think this was literally anthropic did that test they took I can't remember the details here but they uh ran some studies on misalignment and then they took out the
1:08:41training data that related to less wrong discussing misalignment and they ran the same test again and the failure rate went down. So it in fact was some evidence pointing towards it had learned the either the ability or the propensity to do that.
1:08:56Yeah. I mean there's there's the hypersition effect and there's there's like the Luigi Wal Luigi effect.
1:09:01Which is like you are the the more you try to train for it you you create the opposite.
1:09:05Yes. There you go. That's exactly it. In some ways I think the very successful topic is a result of hypersition like the the fact that you wanted this this thing to exist in the world and now it does but like then it also creates the opposite as well like I think I think people who are maybe newer to this space don't remember Waluigi but like I do think it's very very important for understanding that when you train for a thing you also train the the opposite of the thing cuz it's just a bit flip.
1:09:31Yes. I think you know just going back to where we were at like there's a lot more than just mechan that there's value in just having added right so your version of how fast can you measure stuff you have logging you have evals you know do you see other parts of the stack like the inference providers that you use the services okay am I using Chinese model on their home API am I using through certified vendor here am I hosting myself uh what am I doing on the inference engine side there's just like
1:09:59so many levels of stuff that gives you know information and that you can standardize out, right?
1:10:05Yeah. And you also see increasingly in addition to just the basic chatbots, you're increasingly seeing big companies adopting agent platforms where they're building on top of Google's agent studio, etc. that comes with a bunch of like managed everyone has managed agents.
1:10:25And there there's even levels you can host your own managed agents. Open agent SDK or hosted by anthropic or Google does both. Correct. And then these are just ways to kind of strengthen the security guarantees you can make. Uh and in some ways this kind of bread and butter enterprise security. They like they love to host things on their own premises because it gives them really a sense of control. And I think you'll you'll see just like you do in every other enterprise market. If you really
1:10:53sell to the enterprise, you start to compete on some of these security features. And this is also helping AI unsurprisingly. And I think you're seeing some amount of enterprises wanting enterprises are really grappling with the thing that makes agents useful is they're stocastic and the thing that makes them really hard to adopt is they're sarcastic and these are just intention.
1:11:15Leaders come out on different sides of that table in part depending on how much the CEO is trying to get the stock price to go up by saying they're AI native that we must be willing to take the risks. Uh we see we actually see phenomenal tension in the heads of the CESOS of the Fortune 1000 where on the one hand you have a CEO saying we must adopt otherwise we're becoming irrelevant and if we up you're fired and that's kind of like the core emotional attention that we see showing up again and again and again and again and one of the core problems that we
1:11:44solve for them is to take that abstract emotional concern and turn it into a framework in some ways just pride and clarity to to that concern.
1:11:52Is there anything in here? So something I think we kind of skipped over. We talked a lot about agent language model, skipped over world models. Um you guys have voice which is interesting with 11 labs. How about generative media? So you know generating images, videos that's a category that actually has a lot of usage. Is there anything in your current policy? Is it separate policy? How do you see that space? It's like we did talk a bit about copyright. So music. Yeah, music as well. Yeah, I
1:12:20think a lot of the concerns that come up there either relate to uh copyright or there's a lot related to let's call it broadly safety. So like this could be not safe for work or just very graphic materials uh are kind of some of the core things. Uh we have done some work on this. There's a little bit in the standard as well that deals explicitly with that. uh video we've not done a lot in yet and I think for proper production
1:12:49that has still especially proper production without a human in the loop that's still got some ways to go. It's obvious that it's coming but it's very rare that it's like one shot deploy a video to the internet but eventually that will also happen. And we see like you know Luma has Luma agent where it's still pretty human in the loop and that just makes complete sense as the technology matures and over time it will become so good that people will not want to slow things down by having a human in the loop and then uh the need
1:13:17to make promises will grow.
1:13:21Why not just have prediction markets and everything right? It's very EA adjacent.
1:13:25Yes. Uh the core thing is that the people prediction markets rely on public information. There is not a lot of public information. It's just insiders trading on each side.
1:13:39There's leaked information. The core challenge is that often you have private sensitive information and you need to convey confidence and trust around that.
1:13:52And you can of course for some claims like can any model be jailbroken? You could rely on public evidence cuz there'll be lots of people being like well there's tons of studies and actually they all can so that resolves fine. I think that's good for hey this new unreleased methus model how capable is it actually prediction marketers have not a lot to say because actually just no one knows and so I think that's the core place where some of this breaks down is that actually lots of the world's information that guides some of these high level
1:14:21decision is private and often also just not known I think the thing with prediction markets that people like is it's not it's not answering the broad question it's a specific right so will a model do this by this date or is a model capable to do this by then, right? So that's a little distinction there.
1:14:39Yeah. And often the most interesting question if you're say the head of security at a bank, the question you're really trying to answer is will this product, this agent do this bad thing that maybe primarily I care about specifically in the setting that I care about and the question is like what's the closest that information may not exist anywhere. So prediction markets aggregate existing information. this information may not exist and you want something very specific and you're willing to pay for it. That's kind of where a third party audit comes in. We
1:15:09also don't really use prediction markets to figure out whether uh public companies have committed fraud on their books. You use audits. You probably could but the information is just not that available and if so it would be like just trading on bibs. Uh I actually would have been really interesting to see whe the prediction markets 2001 would have predicted Enron going bankrupt and the kind of could you have told could you have sensed from like the the craziness of the CEO or some other trait that they were more likely to cook
1:15:38their books than others or enough insiders leak it that you could also right which is like I mean this that that's the sort of the ideal dream of prediction markets you have liquid markets and everything and then you can compose your exact set of risks to offset.
1:15:54Yes. Yes. Yeah. And I think like prediction markets will bring lots of new information to it. And so the thing is mostly not like which one is it and more like what are the types of questions that prediction markets are really good at and what are the ones where the information doesn't even exist for insiders such that no one could in fact trade on it and needs to get generated.
1:16:12Okay. One self-s serving question and then one open-ended one uh on like the the future of AI. Uh self-s serving question would be so you have your standard right? I run, you know, a large AI engineer conference. Like there's been a lot of talk about us certifying AI engineers.
1:16:28Training programs level one, level two, level three. I was a CFA myself. So I know what that that's what the finance industry does.
1:16:34Would it help if we I had AI engineer level one, level two, level three, and then would they would like work with these guys? I don't know. If you think of the highest level objective as like accelerating secure deployment of agents, then that would totally help.
1:16:48One of the things that happens often now is that folks build agents, they bring it to the to the decision maker and the decision maker surfaces a bunch of security considerations that they had not thought of and now it's not built to spec. Now you have to go and re like add these filters etc. So if you shifted that left like if everyone knew what the spec they were building to if everyone knew the grading scheme.
1:17:10Yeah, that'd be awesome if they were already trained. So by default So you're the grading scheme, right? I don't get to set the grading. You guys you set the grading scheme. You said the grading screen and I think what's uh valuable is like if you can turn this into training programs such that which you're you're not doing.
1:17:24We're not doing that. I think there's value in doing it there. There are others doing I mean not to interrupt interrupt but you know open has their interopic also has like a CCPA thing.
1:17:33Yeah. You know they want they want 100,000 deployed certified consultants. Right. I basically think it's good for we will accelerate adoption if we have more people who know how to build secure agents and we're not working on the side of training people at the moment. I think it's like very aligned with our mission. We only have so much uh attention.
1:17:53I tell you why I haven't done it.
1:17:55It's not like I I haven't thought about it before. It's just being prescriptive right about like well this is what you should know therefore like the stuff that I didn't include is what you don't need to know. Yes.
1:18:04I'm like that sucks. Like Yes. Yeah. Yeah. Yeah. And I think it's like, you know, the the the very interesting defensible thing you guys do is your opinionated 100page report of here's what matters, right? Here's the like prescriptive definition of the requirements you need to be certified.
1:18:23So yeah, and I think that's a choice. I think basically that's a that's a choice and I think that serves some audiences very well where if you're trying to deploy this into a bank or a hospital, etc., clarity of the B those boundaries is extremely valuable. There's lots of other settings where being much more experimental, much more trying it out is just a better fit. And so to me, this makes a ton of sense. Also, you'd have to rewrite your curricula every freaking 3 months.
1:18:49It's fine. I do that. Like it's okay.
1:18:52But yeah, no, for me it's actually like genuinely like the the consequences of getting it wrong and like affecting somebody's career is is a big responsibility. Yeah. Yeah. I I think that's exactly right. And I think a lot of our work actually goes like we don't want to carry we also don't think ourselves able to carry the kind of the true north of what's like secure or not secure but we can coordinate the forum
1:19:20where you elicit all of that. This can be crowdsourced in a way like for your example for what is the engineer certification right? This is a pretty big podcast. There's a lot of takes that people can have and you know discussions that can and people reasonably disagree. So who am I to say like that's a correct question that's a wrong question.
1:19:38Right. So like I don't know vent your frustration to someone that's learning it.
1:19:45And I think there's also you or it matters a lot what the promise is.
1:19:51So if the promise is hey if you've taken my course you will not up. You can't make that promise clearly. You could make a promise of like here's the some important things that everyone should at least know and then you have to fill out the rest there. At least the promise changes. Of course, there's some subtlety in how do you communicate this such that people really get it. Uh but I think it's important to dial in and we have a section in our standard like what is the promise and what is the promise not uh because it's impossible to guarantee that nothing will go wrong. If you need a guarantee that nothing will
1:20:21go wrong, you cannot work with Frontier AI but you can make some claims.
1:20:25Yeah, for sure. Uh cool. uh wanted to end with open-ended. Where is AI going?
1:20:30Um I I think you talked about model stuff, robotics stuff. I just open-ended like where you know what what is what is in the future for you guys? Very near term, we've now started to work with some of the frontier companies in each of the categories that are taking off and we'll we'll continue that work to make sure that we cover all of the use cases that are really taking off. We see a lot of interest once the first one in the market moves. lots of people want to follow them and we think basically AAC1
1:21:00will get to a point where all of the fortune 1000 will organize their risk processes around the standard and you have 50%.
1:21:08No, we do not have 50% today. Uh I think there's some world where probably by end of year we might have representation in our consortium for 50% of the fortune,000.
1:21:18So that's on the Asian layer and then we think the model layer. It's going to be just brings are now surfacing the concerns that are most likely to slow down adoption of AI and then yeah we think robotics comes after that. Um what are you hiring for? What's hard to hire for? We are hiring across the board across go to market and members of technical staff. The people who do really well on our technical team are folks who are really excited about kind of being truly full stack. So let's say
1:21:46when we started working with cursor uh we had never done coding uh tools before. So taking the standard and extending it, fleshing out what does frontier elast look like for long horizon coding agents and taking that problem all the way from like working with cursor and other folks in the space down to like fleshing out and shipping a new version of the standard. Uh so that's like a truly a full stack entreprene entrepreneurship technical people do extremely well at a
1:22:14hard part is building one universal red teamer that works across from Harvey to cursor and everywhere in between that has one consistent methodology one consistent taxonomy of what are the risks and the attacks uh and making we think that's fundamentally the best way to make consistent promises J Morgan is buying both they want to have one framework one consistent way that comes out and the mechanics of making that happen. You get to deal with a lot of the complexity of the real world. I think we have good answers in a bunch of
1:22:43that, but there's some pretty hard engineering problems in and executing that.
1:22:47Can I push a little bit like must you have one? Why not just be like okay look 40% of our use cases are coding agents. So we will specialize in coding agents and that's the that's the one of them and then 30% is like rag. Yes, just do rag.
1:23:01Yes. Uh I think there's some wisdom in that question.
1:23:05Yeah. Um, it depends on what we found that there's a lot of value on is being able to if the decision maker on the buying side, let's say you're the head of risk at a bank and your biggest risk is not in coding or in customer support or whatever the top two biggest use cases, but it's somewhere else. You want to still make sure that that framework has something to say about it to the burning question you have. Otherwise, you'll not earn that trust. Now, it's true that a lot of the burning questions follow where there's a lot of adoption
1:23:35and so great. So do we. So we today do not cover every single edge. But we have a framework that we can add all of these within. We have one global taxonomy of risks and attacks that keeps adapting as like every time a new incident occurs that has never been seen before. Great.
1:23:53Let's go and update the taxonomy. So we bake that in. So I think we have one coherent universal approach. It doesn't mean that we spend equal amounts of time on code and in certain niche use case. Uh we we do spend time where people where people care. We think it's very valuable to have one language.
1:24:12Yeah. Yeah. That makes sense. Um that's that's a that's an important choice. Uh we were going to end actually but I I thought of one final ending closing question which is take this however you want, right? Um let's say one and a half years from now open secret panel of five experts declares that we have reached AGI.
1:24:28Mhm. Do you expect your business to change?
1:24:32No. I think there's some important way I think the the last businesses to exist beyond the labs will be underwriting.
1:24:41Well, there's one there's one job that the labs can never do for themselves, which is to be their own watchdog.
1:24:48There you go. So I I think kind of to the extent you believe this frame of like you'll see hyper concentration like the labs will kill all the startups which uh we can go into the pros and cons.
1:25:01I feel like the labs actually care a lot about this right there was the whole superp position what do we do and we have models smarter than us tier above right models smarter than them training them. So the labs actually think about this a lot they they think a lot about I think there are some of the smartest people on these topics work at the labs. So the problem is not whether they care. Uh the problem is that they will all be stuck in a race where they might have incentive to cut corners and they might have incentive to withhold information from the government etc. And so one kind
1:25:30of feels like eternal truth is that you need an independent third party to go and inspect that data and share information in this case say with the government is more of an incentive problem than an interest problem. I think they're fundamentally all trying to make this go well. What I'm not hearing is like AGI whatever that label means to you to me to to them uh doesn't fundamentally have like a qualitative shift in like you still have to and I think the one thing that would make this a qualitative shift is uh
1:25:58there for some definitions of AGI it will just get nationalized it' be a threat to sovereignty yes and at that point kind of maybe every company is the government the government is every company I struggle to think about that world but at that point you've kind of we I don't think we'll move fastaster enough, right?
1:26:13You know, like we're not we're not set to to do that.
1:26:17But I I have discussed this a lot on the podcast.
1:26:20Yeah. Yeah. Yeah. Yeah.
1:26:21I mean, you know, as far as the the watchdog concerned, I will also mention that because I have my finance background, I often think about the scene in the big short where they talk to like Moody's but also standard and pores and then the lady at Moody's is like, well, if I don't give you AAA rating, you're just going to go down to standard and pors. So, so actually the watchdog is a natural monopoly because if you have race dynamics in watchd dogs then the watchdogs will compete each other to the lowest possible standard.
1:26:47Correct. Uh and so I think what's one of the things one of the reasons why we're very excited about having insurers be around this table is that insurers are the only ones that do not have this generate because they pay the bill. is they keep lowering the prices.
1:27:01Yeah. You you'll find the market clearing and this is not true for movies where uh they don't directly pay the bill if they make recommendations that that are off. Uh so we think that balancing factor is is pretty important and I think it also highlights that there's like no system that's perfect. You need scrutiny of moodies. You need scrutinies of the watchd dogs. Uh for sure.
1:27:21Beautiful. Thank you so much for indulging. This is a beautiful conversation covering everything. Congrats on your success so far.
1:27:28Thanks for having me. Yeah, appreciate it.
