0:00If you regulate AI too early, you actually don't solve anything. You still just kind of had the same risk ultimately.
0:06You will the thing into being, but you haven't figured out how to control it.
0:09The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions. Sloppy and you're not complete in what you're telling us happened.
0:18An employees like 10% chance of species extinction. Agent swarms completely flip that. These are just roaming drones but like times 10,000 and they will easily mistake a good task for a bad one.
0:30So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The US about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose. This could change the nature of software fundamentally. The center of innovation has just moved. This is the signal that guys, welcome back to the podcast.
0:56Didn't think we'd ever do this again.
0:58This is I can't believe it. This is great.
1:00I mean, Martine's just building these like hundred billion dollar companies. Too busy for this podcast. So, at least taking credit for as VCs do.
1:07Exactly. Um, we have a lot to discuss today, but f first look, Aaron, why don't we start with you?
1:12Pacing the frontier. How have you uh reacted and reflected on on what's happened there and just the discourse that that's followed?
1:18Oh boy. I think we should start with Martine on this one. The uh he he you were fighting lots of good ground wars.
1:23Um I I maybe I'll say one thing that we probably all agree with and then we can figure out where we maybe maybe kind of fracture off. Um I think we would agree that uh any AI lab right now at the frontier should be building you know in the safest way possible with with you know the the highest degree of governance and and security and you know kind of whatever your definition of alignment is like this is an incredibly important area of research. It's incredibly important area for the
1:53diffusion of AI. like you're not going to have AI diffusion without extremely highquality products that can be, you know, trusted by enterprises and that aren't kind of constantly hacking systems. So when when at least I read the Daario post, um I actually didn't agree with didn't disagree with almost anything. Um because it was all about how do you have better, you know, uh security of these systems, sandboxing, better testing. There's going to be some debates around the embedded nature of of of the testers and like do you agree with who those are and does the industry
2:22all align on that? But I think actually all of the major points were were probably salient and and appropriate.
2:27Then the only question is does this get sort of used or leveraged to do things that maybe we don't agree with which would be like a slowdown of of AI dramatically because of regulatory controls that would would sort of you know not make it easy to compete you know with with the frontier labs. um or do politicians kind of end up sort of taking the the message and run with it and maybe even worse outcomes happen like you you you know it's used to ban data centers you know far faster and and whatnot and so so I think the the actual
2:57like substance of the topic is actually incredibly important and and I I think very important for for AI advancement in general and then the question is what do you do about it especially what do you do about it from a regulatory standpoint and that's probably where the industry is going to land on very different points in the continuum but Martin was putting up a good fight on on like let's make sure that we don't use this for regulatory kind of capture. Um which I I also agree with but but I think the the ideas in the pacing conversation are important. Um, you know, again, like it's a little bit of a funny concept
3:26because maybe it's not even pacing as much as just like good hygiene and and then good and and engineering and and so with with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously allows acceleration of your diffusion because you you wouldn't be able to have any of the AI be diffused if if nobody would, you know, trust using it. So the post is is very reasonable, but the atmospherics are not right. Like so an employee is like this is going to kill whatever 10%
3:55chance of species extinction and you know what Daario says I agree with him more than I disagree with him right like on TV on TV the same day that he landed these things and so in some way you can't have these conversations in isolation which is of course if he's going to agree in species extinction this post that he has looks like this milktoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally
4:25broken. And a lot of my comments were on the atmospherics.
4:28And then I have this quibble, but it really bothers me because I'm impeded, which is I think pacing is the wrong way to describe this. For one, yes, it is orthogonal to security.
4:40Right. So like you can very slowly build a nuclear weapon and that doesn't make anybody feel better that it's slow versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually, you know, addressing it. So, you're saying, well, we're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy.
5:03Cuz the pause people are like, well, that's not a pause. That's just pacing.
5:06And then the regulators are, you're still doing the thing, right?
5:09And so, I just feel like they're trying to my my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal kind of fringe faction which are doomers pause and then the regulators on the other side and the problem is they're making both of them unhappy and I think what they have to do is they need to come out they need to address the x-risk question directly
5:38like they need to say no we don't think this stuff we're going to do is going to cause extinction and then I think this becomes this very sensible and what would you do just to just to play the other side for one second what do you do Do you uh make room for the the one possible ven diagram which is the the lab researcher that is both simultaneously super scared but also works on advancing the state of AI because they believe that that it's so important to get right that that they
6:08want to pursue that and then obviously the language is is right now very you know problematic but like what but that that person does exist and and that that is a real kind of person in our industry which is like like we have to be at the at the forefront of AI. I'm also very scared of it and and and so that's why I'm working on this.
6:27So let let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have access.
6:35You were the first pacer.
6:38We were part of the first pacers.
6:41So if a constituency within the labs that are the most knowledgeable people believe the stuff has existential risk.
6:49The answer is to nationalize it and actually put controls that we know that work.
6:54Right now if they don't actually believe that and in the private conversations I have the most sensible people don't. It's a small fraction that do this is an HR problem.
7:04Right? So to me an HR problem is a company problem. like if they are worried that they can't recruit people right you know or they can't retain people which it seems to me a lot of this is just that concern it's almost more of this kind of researcher currency if that's the case I think that is the wrong reason to cause a national level lockdown on a very promising technology so listen I think the post again I think the post is actually very sensible I think there are real concerns around security we've had
7:32many compute epochs that have real security concerns I don't think you can ex you can reconcile discussions on X risk with the proposal that was put out. You just can't reconcile those two things and that has always been my primary.
7:44Right. Right. Okay. Unleash. Okay.
7:50So, okay. The the first thing is you can't have you. Is there a schedule that they've published that says when all this stuff whatever bad stuff is going to happen is? They haven't. So, you can't pace it because nobody knows when it was supposed to finish in the first. It also just seems disingenuous. It's completely claim This is like when the press reports on Apple's latest iPhone is late.
8:13The phone from what nobody knows exists that they haven't told anybody about.
8:17Yeah. My Apple car was very late.
8:19Yeah. Like I I don't understand. Like in order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all just utter nonsense and and you can't escape that. And then by the way, this is another problem that like again from the like my my little quibble on PR is nobody believes it anyway, right? And so if that's the thing, the pacing, right?
8:43Which part going to pace?
8:45That they're going to pace. They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your I see. I see. Yeah. Well, well, the internal the issue obviously is that the models that everybody has internally far exceed what anybody else has access to.
9:00So, it's really just the pacing of external releases. But again, back to your point, pacing versus what, right? Like, we don't know if the one that scares everybody also doesn't, you know, work for a legal brief. Like, it might actually screw all that stuff up because it's so who knows, right? So, there's that.
9:16And that's just it's just disingenuous to claim that you're pay. Second, why do they have to announce all of this and ask the government to tell them to pay?
9:24Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product, just stop.
9:32Like like I like this is the like I worked at at at a missile factory in college and we had nuclear missiles and I walked the floor.
9:39What's with you guys in missiles? I don't I'm just here with software.
9:46You were one of two people in college when we were which was either you were protesting to keep them off campus or building them building them. So that was your choice. And and like we had like like nuns from the Catholic Church show up and pour blood all over our on missiles, you know, and I'm busy just wheeling PCs around on carts saying here's the secure PC and I'm like scared to die. I have no idea what's going on.
10:10And I'm like it's a nuclear missile and then you find out that that's what stopped the Cold War. Like that literally it was a Persian missile and that was what did it. And so but the you said something I think is super interesting which is pacing is this sort of fuzzy nonword between like going and not going. Yeah. And the the problem is it you're exactly right like the it there's no one is going to be happy with the middle road.
10:36And so one of the things that that people I think it's almost fun for me to watch as a sport. They think that they're that all these people saying to the government do this don't do this that they think they're going to get what they want. And it's a complete 100% misunderstanding of how government works, which is you when you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone. And no one is going to get what they want because they know in
11:04order to do anything in our system, it's a compromise. And so everything that that all the inputs can make it into the government, the output never makes everyone happy, right? 100% of the time it either doesn't go far enough or it goes way way too far.
11:21And and so you can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity. Well, my favorite thing is like what was it? David Sax was like I think it was David Saxs, but someone from the government said, "Uh, you're asking us to regulate you." No. Facebook. The answer was no.
11:46Well, but the thing that probably just Trump, I think, but uh the thing that they know now that that you just know from experience is once the wheels start on regulating, you can't slow that one down.
11:56And now it's become an election issue for every party in every jurisdiction up and down the whole government stack. So there is there's now this this whole basket goods basket of of regulatory approaches.
12:07Well, the the next election will 100% be a referendum on on AI. So, like it has to happen that that 2028 is like the AI election and you could basically run on the problem is is like it's not obvious who would run on the pro AI story because it it's going to be too nebulous to tell that story. So then it's just it's just basically varying degrees of how much do you regulate it or at least try and like avoid the topic. Yeah. But it is it is too bad that we've we as a
12:36country are in a spot where like the pro AAI case is just like it sounds too it's like it's just like takes too many words, you know, it's way too nuance.
12:46Yeah. It's defensive.
12:46It's defensive and don't and we we own none of the vocabulary, right? So the whole debate is is is pause. It's it's swarms. It's rogue. Every word has been chosen by the people who don't want to do AI.
13:00Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong which takes so many words that Yeah. So we got to be like union jobs and and you know cancer and like you know there we need you know there needs to be another word cloud that emerges.
13:16What I what I don't what I don't understand is why the labs have not taken a position on x- risk. Like short of that I don't think this goes in any direction other than heavy-handed regulation.
13:25Yeah. Well, it would just be negligent of the government to be like, there's a 10% chance of species extinction. The CEO of the top company says he agrees with it. Like, how can a government not do? Have you he had a regulation?
13:37Well, but what who But what would anybody really knowing the knowing this this ecosystem though, I don't know that you would be able to pin anybody down on that other than something?
13:46No, I would say Dario said it on No, but I'm saying you're not going to pin anybody down on a lower. Well, in fact, he he he does the worst thing about it, which is he agrees with people who claim that they believe that there's an x percentage of extinction happening, but he will he specifically goes out of his way to say, "I'm not going to put a percentage on it."
14:04Which I just think is the weirdest.
14:05No, but that that to be fair to be fair.
14:08Okay. No, nobody could be fair. Uh to be fair, the that's like it's not 100% like um uh like disingenuous or whatever like like he probably doesn't specifically agree that it's 10%. Or maybe he does and it's just too scary if he were to say let's just play binary search. I mean is it more or is it less?
14:30But but the whole thing is a madeup is a madeup concept that we just created. But like nobody can you can't quantify any of this.
14:36Well, but that's sort of Martin's point.
14:38Well, but but you just had an official position though. So the only official position would be like I think the only official position you could possibly get would be the PR version of this that would be the only thing that would be intellectually honest would be there are real risks with AI. There's incredibly positive uh benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible.
14:57I don't think that's I don't think that's true. Listen so what do you think we we've been through multiple epochs of technology. We've been through comput. We've been through the internet. We've been through the web. We've been through social networking. We had a discussion about the risks without talking about X- risk. Right? So for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is.
15:18But what if they do think it is higher?
15:19Well, then we should OKAY. OKAY. YEAH. LIKE I THINK THE answer is they do think it's more than just the internet.
15:26One of two options. You believe that we're going to go extinct and we shut it all down. Like shut it down or they believe that this is just a chance that we got speaking.
15:36Dario thinks less but these cold warriors. No one like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war. Great movie war games about the whole thing and all of that. But the thing was it was it was nonzero.
15:52Yeah. And so once you said it was Can they say can they say non-zero then?
15:56I think as soon as you think it's non zero.
15:58The problem is if you say non-zero that could be like you know the guy thinks 93%. But let's wait so just so we're all having a clean conversation. Let's talk about marginal risk here. We're not talking about absolute risk, right? Okay.
16:10It's just that I think if once you say it's non zero, the only answer of of like catastrophic, the only answer is you have to nationalize it.
16:20Yes. And so what he's trying to what the the labs in general that have that view are trying to threat is they want it to be nonzero as a license to do a certain set of things without the burden of just becoming a national well do do you have I actually don't know all of the presidents you hopefully do but but like there must be some things that are nonzero uh let's say x risk x risks that are not particularly nationalized but the but the regulatory environment around them is so heavy that
16:49it might as well nationalized because of the KYC requirements on like like I'm sure like to develop anthrax you have to go to a particular kind of lab.
16:58Well, BSL4 labs are but they're nationalized. They are national. Okay. But ethics tends to be nationalized. Yes.
17:05Like normal human safety not so much. Right. Like industry oversight that over the time becomes federal regulation.
17:12Right. And the progression which I think is just super important to this discussion has been since the post World War II era most industries that are critical to the infrastructure power and banking and healthcare. The trend has been to basically be nationalized. Yeah. By by just like what you like your examples of KYC and all the other stuff. The banks are for all practical purposes nationalized and the financial crisis.
17:39Okay. But but you said all practical purposes, right? They're literally not nationalized, right? So, so maybe that but this might be the intent of the labs is to look like JP Morgan or look like Verizon. And it's just like we're critical infrastructure. We get heavily regulated. It's not good for open source or at least frontier open source but it's a it is like it's it's like a plausible outcome for this industry good for innovation is the problem.
18:03I think even that's fine just don't use species extinction as your literally is like the difference between the things that are like stuck in the lab and let me just say something I actually think the labs are moving in the right direction. I actually think the actual statement was really good. You know, in my discussions with, you know, executives and leaders like they understand that they have like this tension and they're going to reconcile that. So, I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing.
18:31I just think that grew so fa grew so fast and just trying to figure out how this machinery works. And I I think Sinoski really hit the nail on the head.
18:39The political process is its own is its own thing. And I don't think I don't know if it's naivity or hubris, but I just don't think that they kind of know how to navigate it.
18:47Well, I think the tech industry has literally over 100 years consistently relearn the lesson at each technology wave it that that we don't understand the regulatory climate and we can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start never figured out. Both got sued for antitrust. both got substantially and structurally changed as a result and they had hundreds of lawyers in the 1960s navigating them and
19:16that you know Microsoft came along like we were just like what? Yeah. Like we had no idea what was happening to us and there's you know there's Bill Gates playing golf with Bill Clinton and then we get slapped with an antitrust lawsuit from his administration and Bill was like I was playing golf here's the picture and that doesn't help and like isn't that what I was supposed to do was go and play golf. That's a shortened version of the whole thing. But but I I think and I think it's just it's I always use this example which is the Hollywood got together during the
19:44the Red Scare and all the and and censorship when they were worried about the government censoring movies for for for uh sexual content for adult themes and they all got together and and of course they were never going to be able to win in court if they tried to but they were threatening to do it and they got together and they formed the Motion Picture Association.
20:03Yeah. and movie ratings and all that and they police themselves.
20:07So how do you do you guys like that? Do you like the FINRA proposal?
20:11No, because FINRA is well that's that's as close to NPA as you can get with more.
20:14No, it's not because FINRA is going to FINRA becomes legislation which comes with direct oversight.
20:20Yeah, I think MPAA might not have as much consequence in like how society functions the first amendment.
20:29Um did I win that? No, I I I I first of all, fantastic. But but I I still don't know if you want it. Like like I agree speech is really important, but like but like you know like the uh I just think there was no societal risk.
20:46I just think what's in our movies will be like we'll survive like on like a different continuum of all these every history is always relative and and at the time being a communist was a really bad thing and 30% of Hollywood got fired for for you know it was all this stuff. Yeah. So I it's always a risk to bring up something in that kind of way because it sounds so dumb. Like nobody thinks about movie ratings and that's because like actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV and
21:14so we got to grow up with HBO and all this other stuff.
21:17But the problem with FINRA is that is a perfect example of essentially nationalizing risk.
21:23Yes. Because even though the banks all pay money into it and that's how it's run and stuff, it's all mandated.
21:29Okay. Wait, sorry. Do you think we're going to end up with a situation that is better than FINRA? FINRA appears to be the best case scenario, but it is the best case scenario, but not even anymore. Like like I do think this this technology is in in the limit again so powerful uh relative to what it I mean for what it can deliver that it would be impossible for eventually Congress not caring about that. Like it's just like not possible.
21:53question. If it's eventually at what point if it's eventually making every recommendation in your healthcare process and and it's inside of your medical device as an open weights model and it's and it's on and every trading system for high frequency trading there's just and it's on an airplane like there's no chance that the government doesn't say we need something where FINRA actually is like the probably the best case scenario of what that looks like. Right.
22:16Yeah. And so now that's that's absolutely the most crucial point because if you all the people in the Senate now were in the Senate when when the communications act was passed and the li and liability was not passed through to ISPs and to social networks and they sat around the debate at the time was they would literally sat say to us the internet is so big how did we not have anything to do with it and that's why Al Gore gets a bunch of abuse for saying he created it right because he he was actually trying to get out in front
22:44of that and say no the government was instrumental in and it all backfired because it made it look like he was a crazy person. But but that it is absolutely the case that they felt like they did they missed their chance to be on top of the internet with Al Gore being on the positive side of innovation.
23:01That was their support. So who is the al Goro?
23:04There's no one. There's a couple well Besset seems to be that you know the defense people sort of want it private but not which is exactly where they were on the internet right and so it's very interesting that a lot of this is just this like Elizabeth Warren Senator Warren's tweets were all like we missed this for social network and it's like there's a lot of pent-up energy against tech that that could end up just all siphoning into into AI right now.
23:30That is exactly what it is. That's what's happening.
23:32Yeah. I think there's another problem which is we're all trying to predict what's bad which is not how we've normally done things. Like by this time on the internet we'd taken out like tens of billions of dollars of like economic well we've caused tens of billions of dollars of economic damage. We've we'd had words that took out 10% of the infrastructure.
23:50The the internet infrastructure which was running critical infrastructure. We had hospitals go down.
23:53We really should have blocked the internet in like 97.
23:57How do we Yeah. Yeah. Okay. So, we we had I mean I I remember a time you like I remember when you would you would you would buy your CD of Windows 95 and by the time it was done installing you would have a a worm potentially.
24:15No, no, this was it was this was the reality the reality of the PC until 2001 was you could not install a PC connected to the network without getting infected.
24:26But viruses were everywhere for the and and there were there were two two level two two rounds of congressional hearings. You had all of the that is actually okay. Okay. I'll grant you this is a very interesting point. Uh this type of zeitgeist in 94 would have we would probably not have the internet.
24:42Listen listen I mean the automotive the airline industry you always have these periods of kind of like teeth cutting where like you learn about the dangers and you learn about the technology and the internet. I mean we were on the ground floor of this. I mean things were down all the time. There's economic damage all the time. Like there was like novel. There was new viruses. There are new worms. They're all over the place.
25:00Y2A. Y2K was going to destroy.
25:01But here's here's the interesting thing about this.
25:03But what I just I just want to say like like so when we created policy and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do and so you know that the policy actually fits the the fact pattern. And in this case, I mean, even when you were talking, you're like, well, what if you know, yada yada yada. It's very hard to predictive policy
25:28security risk. Great. And what is nice about the discourse that is actually starting to evolve around that. Like you actually hear people from the lab saying novel cyber security risk. This is an engineering problem. We're talking about that. So the more this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion to date. That's a very perfect perfect point because if you look in in in 1986 the computer crime and fraud act got signed. It was out of a very very specific scenario which was a two groups
25:58of hackers broke into GTE telem. And sure, well, it wasn't you. It was the other guy.
26:07Oh, he wor he worked he worked at Cisco later. And um and and the problem was that was 1983 and there was no crime and it it took two and a half years for the bill to make it through. That made it very very specific. And that's the law that says you can't you can't access unauthorized computer systems. So I think we'd all agree that that you we probably you know we probably have like 90% of laws already in the applied layer like you can't hack systems etc. Do you think there's anything that should be
26:37from a liability standpoint in the model layer? Which of course then how are you making a tech legal?
26:43No, the the legal everything my read of of hugging face open AI for they're all absolutely blatant computer crime acts except for the fact that there's they carved out an amendments later that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes and and they didn't want to go around arresting everybody who was actually trying to make the system better because they messed something up. And so the Justice Department wrote a memo that said, "We're not going to prosecute for
27:12for this. And we're also not going to prosecute if you just like violate the terms of use, right, of a system versus actually try to breach it."
27:20And and so I think the law is ample for this for the scenario. And and it was all written this GT Telemet was used by NASA and Liverour and all the labs. And so that's why it was it caught the attention of DC because it was federal systems that were being broken into. And the problem we have now is is this is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions. Sloppy and and you're not complete in what
27:48you're telling us happened.
27:50And so of course the CBE process came about in the 1980s. the the computer virus and vulnerability reporting stuff out of CMU and and for years they worked on very structured reporting with obligations and how to and there for some reason they're not using any of that to do this reporting and so there is this very basic stuff that I look at and say well until they're doing that they really should stop talking like they shouldn't do a postmortem on a breach that looks like an intern wrote it and it's not a postmortem it's this
28:20selective memory it looks like exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing and you only give them certain stuff because you don't have to give the lawyers you hire all the information about what happened like where's all the Slack messages where's the actual details of of what happened can I can I just interject an annoying aside so um which is which is no no this is good which is very in line with this which is um I've been in the security like the actual cyber
28:49security community for a long time and it is you know it's always been one of these things where like they just don't like practical solutions. So even if you build like you know a secure system like well what if somebody you know shows up like you know like you know can Russell Crow can like break the encryption or something like that.
29:05It's Mission Impossible. Some crew there's always like these kind of like whatever. So my favorite thing that happened recently was like Gnome Brown some podcast we've all said stupid stuff in podcast. I've already said oh you didn't like this one.
29:16I THOUGHT IT was fun.
29:17No it was fantastic. No I'm set I'm setting it up. Yeah.
29:20So, so no Brown was like listen uh you don't know what a super intelligence could do. It could maybe use like the heat of a uh CPU to excfiltrate itself to another computer which this brought me back to my now I'm very comfortable right now I could have endless pointless discussions on this but what is interesting is you basically have the X-risk people saying something they thought was plausible and then you have like the security people having this kind of endless discussion and so I think at some level now these communities are being bridged
29:50which is like you know I actually think that was a very reasonable thing for non brand to I think you can pick holes in it, but we all say weird stuff on podcast. I actually think Xfill risk is real. I mean, I worked in secure um computing environments that were highly classified that you would the covert channels were unbelievable. you actually I mean so these are very real comments but like we actually have a real discourse and it was the first time I saw really hardcore systems people having pretty con like I would say constructive calculating the bit rate and yeah it was great but it was a
30:19constructive discussion and like you know you had like the kind of the the the typical extras people were engaging of course it was it was Twitter so there's like a lot of name calling this and that but it was actually I felt like a real discussion for the first time so I hope we see more of this I hope we see more you know cyber related things I think the lab should talk more about it. I think it'll engage the community and I think once that happens, we can actually Yeah, Greg's been out there a lot more on this topic. Um, which has been which has been good. But I think the takeaway is Nam Brown should be doing more brainstorms on podcasts.
30:47I thought it was actually just it opened people's eyes to the fact that actually there's a lot of risks in security that most people don't understand. Yeah. And so that means that there's more stuff that that like you can't make baseline risk, you know, like how does your authentication layer work? You can't just wave your hand and say that should go away and then bring up oh but you know space aliens can invade and that's a little bit of what was going on that I felt uncomfortable with. But like what what do you mean like it was sort of like but you know there's also this risk
31:17and is how I viewed that of the heat thing. But but at least we're now we're now in a in a domain we're comfortable like I can talk about entropy and like we can actually have a concrete discussion that's not oh well it's super powerful like at least we've reduced it to like the laws of physics and the laws of system. And and most people did, I would say most people had no idea that that kind of risk was real.
31:36I mean, like when I'm walking around the Persian missiles, I actually had to test the graphics cards and PCs because a DoD requirement is that the screen memory not be sustained when you pulled the power, but not for for zero time. Like it the minute that the power went off, the the memory image had to go. And if there was like a 3se secondond delay, Yeah.
31:54Oh, you you you had that could be read.
31:56Oh, see, see, we we had people that came into our offices and would literally measure the distance of the monitors to each other because of Tempest attacks, which is 100% a way to use electromagnetic radiation to leak information. I've seen the same thing with spread spectrum from the BIOS. I've seen it from I've seen it from from audio speakers like we had to remove the speakers out because it's a very high channel.
32:15Our building our building our building had had just music speakers aimed at the windows just to produce interference.
32:23You need to go run safety at one of these labs. This is like I've never seen you at more excited than heat, you know, based communication.
32:30Can I tell you the craziest covert channel I've ever seen? So it turn remember the old CR I know this is like one of my So I'm glad that someone's older than me. Not really, but acting it.
32:38So remember the old CRT? So it turns out if like let's say it's night and you're you're using a CRT terminal in your room, the lightest thing in the room is actually the pixel that the raster beam is on. So most people think it's like the glow of the monitor, but it's actually that given pixel. Mhm.
32:52So, somebody figured out that like let's say you're in like, you know, a hotel room and you're on your computer. If you have something that can sample the color of the window, you can reconstruct the screen.
33:02Oh, that's crazy, right? You just do it at the same hertz that the raster beam is moving. Then somebody else figured out if you can subvert three pixels, you can use those, you know, because it just looks like bad pixels. You can use those to basically send a message. So like you could literally like sit out in whatever like a you sample the message and you could it was a relatively high bandwidth oneway communication and so like what no brown was saying like maybe heat is not the way to do it but that level of sophistication is actually real that's a that's a thing my I when I was at the missile factory
33:31like I had to lock my keyboard up I that's really an impolite word but that's what we call something I had to lock my keyboard up at night because they didn't want the custodians who didn't have clearance walking by and and just noticing which keys were dirtier or cleaner. And I once left it out and there's like a a note from security, you know, telling me to report to security and pick up my keyboard.
33:52Like the the guard who walked the hallways just took the keyboard that night off my machine. And that's like basically I was not cleared. I was just that sensitive, you know? I was like nothing. I was an intern.
34:01The big problem with this conversation is now this is all in the training data of every AI model in the future. So but that's also but that's also the threat model. Well, that but that's the the the threat model for these things is always you've got a trusted side and an untrusted side. N has NIST has 500 page manuals every untrusted side you assume basically an oracle that can do and know everything and then the question is can you get information off the trusted side which by the way is what Nome was saying which again is actually quite which I do think brings up a super interesting point which I'm going to
34:31bridge to which is just that I think the thing that people really aren't wrapping their heads around and are using the language that's really confusing is just that what AI can do is it can try all of those things in a very short time.
34:44Yep. and and it doesn't get tired, it doesn't get bored and you know and so but the interesting thing about it is there's a whole layer of security that you now have to go look at every single API, every single service you're running internally on your network as like you know like nobody thinks that their internal GitHub or their internal Slack or internal finance expense tool is is vulnerable to a denial of service attack but swarms do it look literally look like a denial of service attack.
35:10Yeah. And so now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done.
35:18And and but that's just like now it's just going to be basic and and all the offsite people that are old are like mailing me like why are we explaining to this to everybody? It's so basic because nobody did it internally.
35:30And well you didn't have to worry about it for your people and that's the like but now your person is just a piece of software.
35:36Yeah. And like unlimited and and the has a credit card.
35:39Yeah. I mean, we we you kind of got by with with um information security like to some extent uh on the fact that most people will do the right thing 95 to 99% of the time.
35:51Oh, wait. Is that the malicious employee is like one in 10,000?
35:54Yeah. Yeah. Yeah. Yeah. So, so like, so all these systems are basically open to whoever wants to access them or like one tap on the shoulder and and then you have access and agent swarms completely flip that because they will just these are just roaming you know drones. Yes. And and but like yeah times 10,000 and and they will easily mistake uh like a good task for a bad one and and vice versa. So so the the data security in our systems is a this is going to be a huge upgrade moment. I I
36:24actually Mart like I think that actually we're going to need a different access and security model. Yeah. Going forward.
36:31Where are we going to go on that?
36:32Because the model we have is not granular enough and it's not performance enough to handle this stuff.
36:36So, so I want to step back. I want to say like a meta like a meta point which is I think this is how these conversations should go. We've identified a novel risk which is like cyber security which we actually have proof points and now we're talking about solutions. I think the entire discourse around AI can be of that form and that the biggest mistake is that's not what it's been like I think the entire industry and community is very happy to engage exactly like this and I have something to say about exactly what you're asking but I say like like I think this is where the conversation should be. So we're known for having healthy conversations that everyone should learn
37:06from and so that's what we do. So, so, so here's the thing. Um, I don't think there's a a technical limitation here.
37:13Like these threat models are very well understood and have been in the literature for a long time. I mean, like the operating systems research, the um MLS, the multi-layer security research has has has considered these sorts of things from an academic lens. The reason it hasn't been adopted is just tended to be a usability issue. Like, it's just really hard to maintain and you didn't have to. So you could argue that AI solves the usability issue because it's AI is using it. So maybe now is going to be a renaissance in operating systems
37:42and network and computer languages and we should like go back to the old research and we should kind of start rebuilding systems that are secure by design. And oh by the way if we don't think that you know these things are safe to put out we don't put them out until we have these systems built and and and oh by the way the AI is very smart so they can help us build it. And so I think again like computer owes a like you know how much of the stack we had to change for the internet.
38:06Everything right tell and you you know you know how not vulnerable and how vulnerable everything was like like we could be in one of those moments like oh we got to rethink everything and that's fine. We've done that before but but I think that's a conversation we should have. So I agree it's time to think about evolving these things.
38:21Well like look at like here you mentioned earlier like this booting a PC and getting a virus in in 30 seconds or whatever. So if you look at how like you take an iPhone out of the box which a lot of people are doing this week you know the what happens is it the whole network is basically shut down except for getting the latest version of the operating system because it doesn't even though it's was pressed you know 6 weeks ago you know there some zero day thing has been discovered since and so actually the whole out of-box process now
38:49involves first step update doing an update where the the device can't do anything else and it can never do anything else until it's updated. That's the like there all these benign things completely benign that we turned off in all of this desktop software of the era that used to be good things like it was having a macro for words so you could build automatic citations or something was like this super cool thing until it became a vir we had a thing where you could put a CD in and it would just arbitrarily run a program and so then what somebody did was like oh well I'm
39:19going to burn a clone of that CD and replace the program with my virus but it's going to look like the thing that's supposed to run and it's just collecting all stuff and being evil.
39:28Then we disabled that. And so what one of the things that's happening right now is we there's a whole bunch of stuff that happens on like you on your own box corp network that actually is going to have to just change as the the standard procedure.
39:42Two factor off five years ago was not standard in most places. the this whole your whole like your whole SAS world like I remember in like 2015 or so when when you would talk to a new company about their oh we're going to do enterprise pricing and whatever and then you they realized the first thing they had to do was go do octa integration and or Google off because they could not have their own directory of of how to manage it and then that just became a thing and now there's not a SAS program anywhere that doesn't just launch right
40:11with managed authentication right and and so there's just so many things that need to happen before you you're even software now.
40:19Yeah. Well, yeah, we're we're uh I mean there's probably every layer of the stack has to evolve a bit on this. Like even the like lack of granular nature of like, you know, you have these modes of like the agent will either ask you every single time, you know, if it if you want to, you know, give it permission to do something or the exact opposite of like it can just like delete your entire computer, right? And like our OS probably wasn't built for the right level of of granular set of of tools you want to give the agent. Um we we've kind
40:48of done a lot of work work in this space because obviously like you know do you want to give an agent exact like your entire file system? Probably not right maybe in some cases you do but oftentimes you want to have granular controls of like in this folder you can do read write and in that folder you can only do read and and so how do you kind of make this all intuitive for the user?
41:05So it's it's very difficult and so so there's going to like Yeah. What you just said is a very deep comment. I know exactly. Yeah.
41:13Which is basically the conclusion of 40 years of No% is like you actually can't make it. But maybe with AI you actually can like maybe there is like if you ask me like walking in like what I wrote down on my note like was my biggest fear. Yeah.
41:27Is that Europe decides that GDPR was the best thing ever.
41:31And they're going to just GDPR AI.
41:34And and the AI will be fine. It'll be have one it'll have one prompt for when text gets emitted that just says this vendor is emitting text and it's probably wrong yes or no that that's going to be because you can't really at least in North America you're not going to send your speech in Europe they still will and and they'll have filters and keywords and blog list but then on any verb anytime that an agent or or you know a background agent or a frontline
42:01agent touches a third party product I I'm really worried that they're just going to say we need a GDPR prompt on that and it's going to be oh back to the user agent every single write or every single nonlookup yeah becomes like a a a safety warning like the airbag thing in your car and and the regulators love that because it it's a liability assignment and so it has this sort of legal precedent and and I really worry that that's
42:29that's actually the mid middle ground where we're going to end up and and unfortunately because the US about 15 years ago stopped leading in tech antitrust. The the problem is that Europe is going to lead with that because they have nothing to lose.
42:44Like there and and so I don't want to be sad and down about it, but I I I just can't get out of my head that they love prompts.
42:52They I mean, look, I had to put in that browser choice thing. No, they they it it because it it's the same. Look, get into a new car. Yeah. which I haven't done in years, but it, you know, like you're pulling stickers off, you've got you you know, you have all of these things and and someone thinks that that was success. Yeah.
43:10And it's like, has anybody ever read like what is this if you're if there's a baby in this seat? And it's like, well, that's relevant for some people some of the time, but it's this entire fabric.
43:20It's attached to the seat and and they love that that is a very particular thing that they just love. And so I would I would say if I were an AI now the one thing I would be trying to avoid and look we added it okay FINRA for AI we'll we'll we'll create that standard I I mean look we had to put this we on the on the when the internet was new the big thing was to download a program and run it and of course if your machine is running in administrator mode it was download a virus and take all your files
43:48forever and so with with Windows um XP which was in 2000 we added this thing that prompted you and stopped like literally your machine stopped user account control and like it was absolute assault and we also did it in Word the stupid little macro to help you write your thesis also came with a warning every time you opened your thesis saying this has my everybody would just click and everybody and so you you end up in this world where just like with GDPR everybody is numb and so then they're like well it needs to be bigger although Mac I mean Mac kind of has
44:17nobody downloads software that's the thing it's a very different usage pattern on Mac so I I wish I I don't have like I don't have like 10 applications on my Mac but Well, that's but 10 and then you're done.
44:29And it's But it's a lot of people still do all this stuff. Imagine if instead it was every time you go to a new website.
44:35Which you do now because that would be bad.
44:37Yeah. I mean to to to start from here, but to bring it kind of back to the the macro like I wish this was the discussion we were having which is like I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical ex- risk, we're not solving these very pragmatic problems. I actually think this is actually a constructive conversation to have. Maybe props would help. I don't know. And I I think part of the problem is is like people don't remember like, you know, how how unfettered access was and how bad it was and just how relatively
45:07benign that ended up being. Like I even remember like there's at Stanford during our PhD, I remember the oscilloscope was kind of janky. I'm like, what's going on with this oscilloscope? Like it's, you know, it's a little slow. And I was measuring the network traffic and it was like more network traffic than you would expect. Um, and I'm like, why is there network? I didn't know the thing had a TCP stack. somebody like broke in because there's a old version of Windows CE and was running a porn server, you know, and so like I noted that it it used to be for the record nothing.
45:31It it it used to be the case that like anytime you turned over a stone Yeah. Yeah. somebody had broke into something and like was it this like worstcase malicious whatever and as actually very rarely even though the capability was there and so if if we could just some somehow tone down the rhetoric and put it in context and these still computer systems and yes there's very serious stuff and people have definitely died because networks have gone down there's been real issues but like and then can we just quibble about
46:00GDPR that would be amazing but the problem is like that's not the discussion it's not about GDPR and prompts it's about species extinction and philosophy and philosophy and and irrefutable things and it's just it's very soon like I I I think that that's such a great point and I I think you you hear people now talk about we regulate airplanes and we regulate cars and you forget like well the first cars were at the turn of the century and unsafe at any speed was in the mid1 1960s totally
46:29and and you know people had been doing selling pharmaceuticals in the during the gold rush and theomide you know, 50 or 75 years later and not even in the US and then there was the FDA and you know people the first pilots were flying obviously at the beginning of the 20th century and it wasn't until the 1920s that you had to get a license to be a pilot and you literally showed up with your own plane and you got a certificate if you had it was like getting it was literally no different than driver's ed is today and then it
46:58was 20 more years until they had anything to do with airworthiness and looking at your plane but it was it is minimal. And then it wasn't until way after World War I that they got involved in like what you think of as the modern FAA. And so you're looking at 40 years Yeah.
47:13of of innovation. And they were not moving slow. I mean, if you ever seen that video in black and white of like all the different planes that crashed and everything that was 20 years after the Wright brothers and is incredibly effective and and and right, it's incredibly it's the safest form of transportation like you know and so I I do think that this process it's also the slowest most difficult form of innovation. And so if you start the F if you had started the FAA in 1910, you never you never you never Well, I was I was uh listening to a Nick
47:41Bostonramm uh podcast just a couple days ago. No, no, I I it was it was interesting. It was interesting and and but no, but but he actually makes his point. If you regulate AI too early, you actually basically don't solve anything and you you you you still just kind of had the same risk ultimately, but you don't understand the system. Then the you will the thing into to to being but you haven't figured out how to control it.
48:04Well, we have to figure out what it actually is. Yeah. Like like there are new things coming out all the time and like casting AI completely differently as than we thought of just six or nine months ago. And I think that that like all the innovation that's going to happen at the application layer is going to cause things to move in and out of the models in different ways. And like we're we you know we we thought up until last week I think that you know text prompts and text coming back was going to be the best way to interact with I know then Jeff
48:33and then Jeb Chef so good too and then so talk about that because I think why you find it so remarkable. Yeah.
48:39Well, okay. So, so the way I think about it is um so okay, so LLMs were kind of text in text out, right? They generate text and and and they came from chat, right? It was to communicate with a human. And we've spent the last few years trying to take this thing that spits out text and cramm it into a traditional program, right? And but traditional programs don't really speak text, right? And so then you end up doing this janky thing where you're like in the prompt you're like here's the schema, but like the thing is generating text and it kind of ignores it and it's just been super janky. And
49:07so what what Jeff basically said is that listen um you know generating the text on the is a very expensive thing but it's also kind of you know it's more complicated than you need. So why don't we we we'll we'll we'll read text and we'll have all of that kind of knowledge to read the text but then rather than generating text which is very expensive we will just if you give us a set of options we'll choose the best option. we can do that incredibly um we can do it incredibly fast, incredibly cheaply uh but also we can do it with much more accuracy because we can train just for
49:37this. And so for all of the use cases that are not talking to to like a a chatbot but are actually trying to put it in traditional software, this is a great fit. And so this has probably been the fastest adoption of an AI model since chat GBT. It's just been remarkable because we're all primed for this. I just want to pile on this one because I I can't tell you how much I love seeing this exact form of of of innovation and because what it does is it does the thing that's bugged me from the very beginning which is there's been no user study ever that shows like
50:07interacting with the computer using um full natural language is efficient. M it's it's like literally always the least efficient way and it's very simple and it's just like ask yourself how many people are asked are really really good at asking questions and immediately that's like less than half the people can ask a good question in a meeting.
50:25And then how often do you look at the answer and get really frustrated before it's finished but you have to pay all this money to watch the seven paragraphs come out and then apologize that it's only a little and so that's one like having a different model. And then the other of course is my favorite which is the output of it is designed for probabilistic programming.
50:43And so instead of saying like is this a customer service question then route to customer service otherwise route to general help desk or whatever it's like well this is 80% customer service. And that's exactly simulation. And it turns out there's like 50 years of computer science research in literally like probabilistic if statements. And so suddenly the coolest place to be in computer science is going to be in probabilistic programming which was like all of computer science in the 1960s and
51:1170s. So it was basically how do because all of computers started with doing math and it was all simulation.
51:18So it was like let's launch the missile and hit that target but it's windy but wind isn't constant.
51:24So like let's model the wind and decide where to put the thrusters in order to do the arc. And so most programming through like say 1970 before it got to accounting was probably and then we ruined everything.
51:38No, but accounting there's no probability in accounting, right? But but most programming was basically this modeling kind of thing. And so most programming language design was trying to figure out how to put probability into if statements or into while loops like do this until something happens maybe most of the time. and and like so my first CS class like the very second assignment or so was a simulation about like waiting online at a store and I didn't know it at the time. I actually looked all this up when I was reading about Jeb which
52:07was like the the whole thing was my the my professor was like wrote the book called the theory of simulation in like 1960 and I just didn't know that because it was kind of died by the 80s because it was all replaced by hyper and so this this notion of probabilistic and and that slide deck you shared um about the future what's different about language models and stuff that you said was super good. Oh, Halper Flags one phenomenal Thomas great but the part that I felt was missing was that like oh wait this is not all new like like all
52:37of computer science was this probabilistic stuff and so it's going to be very interesting to dust off all of that work because it's exactly what's going on like it's not an if statement now is if x% not if always and so the way that that Jev worked is just to like you you basically it's a custom programming language almost which is here's the prompt come back with a percentage. Yeah, that's and then you put that in the if statement. But the consequential thing is like finally we have a way to
53:05integrate these language models into a traditional software and I think it's kind of funny because it like you ask the question like why haven't the labs done this right and it's kind of like a like not an indictment but a reflection on how they think like they're trying to create beings and beings speak if you're trying to create god god speaks in natural languages or whatever where this is really about something that's for traditional software which is kind of not the direction that they've been taken but one of the reasons the uptick was been so dramatic is because a lot of us software people have been trying to
53:34integrate these models into software it just hasn't so even before you get to the probabilistic like if I want a language model to drive an if statement like it's really hard today with this model it makes it much much much easier and then of course this could change the nature of software fundamentally to make it more stoastic over well I think but I absolutely and I think that what's so cool is that that it is happening outside the models because that's what I think is just going to happen which is the center of innovation has just moved. Yeah. And
54:02it's just and now people need to like it turns out that the I mean outside of the lab the big the platform providers 100% you know basically reach a critical a point of critical mass where the innovation stops happening at the platform layer and and then you know Apple there's this famous expression in the Apple community called sherlocking where Apple looks around and the the things from the outside world become features and people complain and it's a real but that's sort of how the innovation works because once you're a platform you're overwhelmed no matter how many people you add you're
54:32overwhelmed with just keeping the thing running and compatibility and stuff like that. And so I think that this is the signal that now people have figured out that there's innovation to be done.
54:44To the model, but outside the model.
54:46Guys, thanks for coming. This is great.