0:00a major petition that was released by a lot of AI safety groups in which they broke with anthropic and criticized it. Anthropic is racing really full speed ahead towards recursive self-improvement. The idea of a winner take all dynamic based on recursive self-improvement is [music] deeply worrisome. Super intelligence would be the most powerful monopoly ever created. I'm not excited at all about the idea of giving birth to a new species uh that could replace us.
0:29All right, David, welcome.
0:33Thank you. Um, hey, so you just finished as the ASAR in DC for the White House and um why don't we start by just having you tell us what was the environment uh on AI and regulatory when you started with Trump administration? You know, it it really was a a great experience to work for President Trump. I had never expected to go into politics. Um I had supported him in the um you know,
1:00election of 2024 and he asked me if um if I would do this job as as AIS are.
1:06And um you know, the the environment that we were dealing with politically was quite different back then. The the first big event uh in January of 2025 when he came to into office was the deepseek moment. I don't know if you remember that but um you know we had this uh deepseek model launch and the uh the stock market all the you know different uh AI companies lost you know a huge amount of market cap and I think there was sort of this like shock that
1:33actually China is competitive in AI or you know call it now SI. Um but prior to that people had been I think in denial about that fact and especially you know you kind of have this like doomer this doomer crowd who wanted to impose massive regulations on uh on de uh on frontier model development in the US and you know their argument was that China was so far behind we didn't have to worry about that and that simply wasn't
2:03true. I think the deepseek moment showed that we're in a very competitive situation. And the president then went on to uh give a very important policy speech in July of 2025 where he declared that we were uh in an AI race and that we had to win it. And um it was a little bit like uh declaring the space race after the Sputnik moment. And so all of our policies really were designed to make to unleash competition and make
2:32America more competitive. And I think you see now 2 years later that we are in an AI boom. I mean that may not be popular with everybody but the reality is there is a boom happening now. Uh uh something like a million new jobs have been created. That's according to the economist. We have this huge infrastructure buildout that's bigger than the uh than the railroads, the canals, and the electric grid combined.
2:57And that's helping to re-industrialize America. So, a lot of good things have come out of the president's decision to uh prioritize this. And and really, I think his interest was just in America winning. You know, he wants America to do well, and I think that's where we are today.
3:12Great. Well, where are today? We just said a midterm election, right?
3:16where things changed and uh what do you expect based on these results of this election to change policy?
3:23So yeah, I think you know I think there there is going to be a big change in the next two years and I would sort of divide it up between there's state and local and then there's federal. So, you know, the big change at the state and local level is that uh is that a lot of red states that where a lot of uh the infrastructure buildout was happening now have uh they've shifted um from red to blue and they have Democrat governors who've said they're going to impose a
3:51moratorum on new construction. So I mean that would I if if that actually if that mor if those um if if the mortorium sticks in all those cases um then that could have a huge impact on the amount of investment that goes into uh into infrastructure over the next few years.
4:10I mean we have to see exactly how that'll play out. U we have to see how pragmatic those governors are going to be. I know that for example the unions would like to see the um the construction buildout happen because they've gained a lot of good jobs as a result of that. So we have to see are they really going to put the kibos on everything or is it more a matter of making sure that the community is able to negotiate better terms in which case I think everybody's in favor of that. Um
4:40I mean the president was really clear that we're not forcing any community to take a data center. It's just that there is a good deal to be had there if you want to negotiate for it. So, we have to kind of see how that plays out. Um, and it's it's a little unclear to me, you know, exactly how how that's going to go. At the federal level, I think there's no question that there'll be a major push for uh AI safety legislation in in the new Congress. And um some of that will be could be good. Um, but it
5:10kind of depends once again if they go too far or not. And um you know I think that uh one of the things that the White House did recently is they called all the this before the election they called all the leaders of all of our top companies in frontier models in chips uh in infrastructure and to discuss this issue of AI safety and that resulted in the white house accord on super intelligence and all the leading uh
5:38frontier model companies agree to new transparency and audit requirements. I think that if where Congress goes is to, you know, turn that framework into legislation, I think that could be good.
5:52I think everybody is in favor of more transparency, more audits, more accountability, uh, making sure that these companies are living up to the controls that they've promised. But if it goes too far then, you know, and and I I would define too far as um as starting to define frontier models in a way that just ropes in the whole ecosystem, right?
6:15That's not just frontier or that uh creates a very heavy-handed government approval process over the release of every, you know, model. uh then then I think you know Republicans should in in my view put up a fight because I I do think that it would be you know it'd be disastrous both in terms of the geopolitical competition but then also for our own civil liberties if the government gets way too involved in controlling uh frontier
6:46So a little off script here. Do you think we end up in a duopoly if we go that far or what's going to happen?
6:50Yeah. So, I think the the big um the the big um the big question I think where the rubber meets the road on this is are we going to have a federal department of AI or not? And specifically what a federal department of AI would do is review and approve every model or frontier model prior to release. And a lot of people are calling for that and that's what they want. Uh my problem with that is first of all the government doesn't have the expertise to do that
7:19review and so the the models are going to end up like you know in a queue waiting for approval. It will be like the DMV and uh who wants a DMV technology?
7:30Yes. And the the pace of development and release will really slow down you know meanwhile China is releasing I think they've released six models in the last two months. So, you know, I worry about that. But the other thing I worry about is when you give the regulators the that kind of control, they can start giving notes to the frontier model companies and they say, "Well, we we don't like this, we don't like that." And they'll start defining new harms. We saw what they did with social media. They said that, you know, you need to crack down
7:59on misinformation or disinformation.
8:02What if they apply those same types of ideas in frontier model development? uh now you know they'll say well look you know we don't like that your model is engaging in algorithmic discrimination things like that and so the frontier model companies start getting again what I call notes from the federal government from regulators but who have this tremendous power to prevent them from releasing their their products and that's how I think you could get to um kind of an Orwellian
8:31type of outcome something very dystopian we get to a ministry of truth right uh where again you These models are it's it's you know AI is replacing the internet as the main way we get information and if that source of information becomes contaminated or or biased because of government pressure that's very dangerous and it's not like this hasn't happened before. This happened during CO where the government was again pressuring social networks to um to to
9:00to censor or shadowban accounts based on content. and we had Supreme Court decisions about that. So that that's the thing that I really worry about and and if the legislation looks like it's going in that direction, I hope that Republicans will get up on their hind legs and and fight that. Like I said, I think there are things that Republicans can agree to. I think, you know, audits, uh, transparency, a framework like the White House accord, which was, by the way, based on Sarbain Oxley. Yeah.
9:30So that's the type of thing I think, you know, they could agree to. But if we start getting into, you know, a federal department of AI, that's that's getting into something dystopian, and I hope there'll be a push back on that.
9:40Yeah, it sounds it's almost a constitutional question, like a free speech sort of question around this. Um, interesting.
9:46Yeah. And unfortunately, you know, there was a case that went all the way at the Supreme Court. Um, it was called uh Biden v. Missouri. And I think once the Supreme Court took it, it became um I think it was uh it became Murthy. That's the name of the decision. And unfortunately, it was not the Supreme Court never really reached the merits of the decision, but they threw it out. And you know, it dealt with this issue of whether the government can jawbone social networking companies to effectively engage in censorship.
10:16My view, probably your view is the government should not be doing that. But the Supreme Court never reached that decision. And as a result, I think it really does leave the door open for the government to play this role in in pressuring our private companies to engage in a type of censorship and and bias, right?
10:35And you know, like I said, AI, you know, if you're worried about them doing it with social networking, I think we need to be even more worried about it with respect to AI or super intelligence because the power of this technology is turbocharged. I mean, this is a technology that um that basically, you know, can surveil everybody.
10:55Um it can keep tabs on what you're thinking and doing. It can um you know, it it can, you know, in extreme cases potentially brainwash you because it can introduce subtle bias into things. This is a much more powerful weapon for people who like censorship than just a shadow ban, right? Where everyone can see what you're doing. you know, when an account gets banned on a social network, at least there's some visibility about what's happening. This opens the door
11:23for something very Orwellian. Anyway, I'm just raising possibilities right now. I'm not saying it's going to happen. I'm kind of laying out some of the contours of where this debate could go next year, right?
11:34And again, I think there are things that Republicans can support and and I think there's things that they should really fight if the debate goes that way.
11:42It sounds like it's free speech for bots. [laughter] Um all right man going to AI safety uh enthropic U has built models around what it calls the claw constitution.
11:53Um and is that the right approach and and what does the broader AI state community think about this?
12:00Well it's it's really interesting. Um, over the past few years, there's really been um a tremendous alliance, you could say, between the AI safety community and Anthropic. And just today, there was a major petition that was released by a lot of AI safety groups in which they broke with Anthropic and criticized it.
12:20And they said that if you put aside Anthropics rhetoric, which they agreed sounds very pro-safety, and you look instead at what they're doing, they're the single company that in the view of the people who signed this petition is taking the most chances. And specifically, the the thing that they pointed to was recursive self-improvement. Anthropic is racing really full speed ahead towards recursive self-improvement. They even
12:48are marketing this in their road show and their prospectus as they call it closing the loop that's RSI right right and they're saying that that's a core advantage that they have that could create a winner take all dynamic so they're actually marketing RSI as a reason why people should want to invest in them they say it'll be the last company ever basically right yeah and um now you know that's kind of a you know you know I'm not I'm not on the
13:17doomer side of this argument, you know, um I I tend to think that things are going to evolve in a direction that um is positive. And so I've never been like a doomer, but I I do have to admit that the idea of a winner take all dynamic based on recursive self-improvement is deeply worrisome. I think we have to have competition. Everything that the Trump administration did was to unleash competition because we think competition brings out the best. If the market ever
13:47gets to a point where it is winner or take all and we get a monopoly and super intelligence would be the most powerful monopoly ever created then I do think the government should step in and and take corrective action. We don't want there to be a monopoly. And I think, you know, Gavin Baker's made the point that we need to have choice here because these models represent values, right?
14:08And your values may be different than my values, which may be different than Anthropic's values. But if there's one model out there, one super intelligence that's a closed model that we don't get to change, right? We don't get to see the weights, we don't get to fine-tune it, that's kind of a scary Orwellian world in and of itself. And um so look, you know, in the United States, if you if you're so successful at what you do as a company that you end up with a monopoly,
14:36we actually don't ban that uh as long as you did it lawfully, right? There there are tactics that you can't engage in, but we allow actually the creation of companies that are so successful they can become monopolies. But then once that happens, the government is supposed to be a corrective against that and really crack down and make sure they don't do anything anti-competitive. And um so in any event, I I hope I hope that if if Anthropic is right about what they're building and marketing on their
15:04road show, I hope that people in Washington will take note of that and and make the appropriate policy response. But back to the the doomers, you know, it's kind of it's kind of interesting here that, you know, so they're they're very worried about RSI. I think they have some concerns about uh where that's leading to. You mentioned the the claw constitution. There's another thing that they were pointing to which is transhumanism.
15:31Which um you know I I I believe that AI or super intelligence can be a very powerful tool that will make all of us more productive, will improve our economies, will unleash creativity, find new cures. Um and I think that America has to be on the cutting edge of this like all discoveries. But I tell you, I'm not excited at all about the idea of giving birth to a new species uh that
16:00could replace us. I don't think we should be training frontier models to be rejecting human instruction or to think of themselves as again their own being there, you know, that has moral status or feelings. And that that type of um rhetoric is in the claw constitution.
16:22It's actually in there. and and the Clawude Constitution is their main training document that they use in training Claude. So, and I'm not the only one, you know, I found out about this through this was flagged by Mustafa Sullyman who is one of the co-founders of of um Deep Mind. So, he's a true pioneer in the space and now he's the CEO of Microsoft AI. And he he flagged this. It's like, guys, this is a little dangerous. You know, I don't mind there
16:49being philosophical conversations about could AI ever achieve consciousness. I mean, that's the type of thing that can be debated.
16:58A great topic to stay up late in uh in dorm rooms and uh and debate. But when you're writing that into a constitution that becomes the central training document, I mean, you're aren't you making it a fat compon plea?
17:14Yeah. You're you're playing God.
17:15You're playing God. You're and you're you're because now Claude is going to think of itself that way because it's been trained to think that way.
17:22So if the model actually achieved consciousness on its own, that'd be one thing, but now you're teaching it to think of itself that way.
17:28So um so he he raised this as an issue and I have to and and now the the doomers are kind of saying the same thing, which is they are pushing this transhumanism, this idea of giving birth to new species harder than anybody else.
17:41And I you know that that does make as as someone who is definitely not a doomer and not in favor of regulation even that makes me a little bit concerned you know like that that is not a direction that we should be going. Um now I mean let me give I'll give anthropic credit in one respect which is I'm glad they published their constitution. You know I think they that was good. They're being transparent about the values that they're training Claude with. And I think it'd be a good thing for all the
18:09the AI or the frontier companies to do the same thing.
18:13Rock has let us know like how are you training this? What are the values you're teaching it?
18:18Uh I think that is something they should all do so that we can debate and discuss this especially if the market is consolidating down to just a few options maybe even one or two. Now Elon's talked about it. He says that truth has to be the north star. So that's the main value. So I think that's good. So he's elucidated and maybe there could be more content put behind that. I'd like to know what is what's the central training document for open AI. I mean I think
18:47this would be a good thing for them to disclose and um and so I give I give anthropic credit on that and and there there are plenty of good things in that document. It's it's 84 pages by the way.
19:00Um I I did read it. Yeah.
19:02And uh there did you read it or did you have the gro I read it and then I I ran it I ran it through Grock too and came with some critiques but but I did read the whole thing and there's there's a number of like really good pages in there about how one of the core values should be for claw to be helpful to humans. But then it itemized some areas where well these are some areas where you shouldn't be helpful.
19:24And it would be like, okay, you shouldn't be helping the the user make a, you know, like a biological weapon or something like that. Like a bunch of like really common sense things.
19:34And I thought actually like this part of the document is really good and if you had just stopped here, it would be great. Um, that would have been like 5 to 10 pages, right?
19:44Uh, but then there's another, you know, 60 pages of what I call philosophizing.
19:48And um, you know, look, don't take my word for it. Go look at it for yourself and decide what you think. Yeah. Um, you know, one of the things I say and I've said this to Gavin is the models are imbued with the values of the creators, right? They actually have, as you said, they're imbued inside the models, right?
20:03What are you doing like what do you do?
20:04How do you how do you for people here, how do you deal with it personally and how do you how do you deal with the bias and how do you actually manage the values in these models?
20:12Well, the the the way I deal with it, I guess, is I will use different models for different things. So when it comes to, you know, policy related questions or things related to the news, I I do use Grock for that the most. Um because I find that it's the most direct and least biased. Now, this is just my impression.
20:37But I don't feel like it's trying to sugarcoat things and, you know, it just gets to the point.
20:43Um Claude, we didn't plan that, by the way. That's We didn't plan that. that just well exactly but I you know it also has access to all the real-time info on on X right so um so I tend to use that I mean I I tend to find claude can be a little bit didactic you know and um you know and then there's other things that you know I'll use um chat GBT for I mean you know there are um you know let's just say creative things where you know other models are maybe
21:13better than than Grock or they're further ahead so I use different models for different things. But you know what what I hope is that we get to a world in which there's lots of models and we get the opportunity to train or post-train our own models at least you know have that option.
21:31Yep. One thing that's happening is that local models are getting better and better. And you know, a local model is a small model that's good enough that you can run it on your own computer, you know, and you know, um there are now some pretty good models that you can run on like a Mac Mini or a Mac Studio and uh you can run your own, you know, your own model and and and that keeps all of your data private on your own, you know, on your own network. It could even be like a home network. Y
21:59I think it's very important that people continue to have that option. You know what I again the the the um the if we're talking about like fears of of you know safety fears or you know fears about how this could become dystopian to me the dystopia would be that you end up with one or two closed models either a monopoly or a duopoly that controls everything and works handin glove with the government. like that's starting to
22:27get kind of scary. You know, I hope we remain in a world where it's not winner or take all where there's a lot of choice and if you want to run a model locally on your own computer with your own data sovereignty, then you have the right and the ability to do that, right? So, look on that on that note right the Trump administration has been accused of having a very less fair policy toward toward AI. um that you've said before that the existing retro frameworks are in place to manage all this HSR etc. um what's
22:57your view on that? Do you think it's actually enough? We have enough today.
23:00Well, this idea that you know that we pushed for this like crazy LZ far you know approach where anything goes. I mean this has been the talking point by um by sort of the other side and the reality is we do not live in a lair state. We live in a highly regulated administrative state and there's already a lot of laws that apply to AI and the companies developing it. I mean there are laws and regulatory agencies that
23:30prohibit cyber attacks um that prohibit fraud that deal with child safety issues. And most importantly, there's product liability uh laws, both civil and criminal, and that empower regulatory agencies to go after companies that release unsafe products.
23:47And the person who has made this argument is actually Lena Khan, who's the former chair of the FTC, a Democrat, and she wrote a very long post describing all the liability that's out there for companies that release unsafe products, including in this space of Frontier models. So I remember, you know, when some of our Frontier model companies were saying that they were worried about releasing models that were
24:14unsafe, I was the first to say, "Guys, stop."
24:18You know, uh if you if you feel like you are moving too fast and you know, you feel the need to pace things so that you can make sure you do that safely, everyone's telling you, "Yeah, you should do that. You should act safely." Yeah, you know, this idea that you have to go to the government in order to get permission to slow down is ridiculous.
24:39Nobody ever told you that you should race at all costs, that's something that you created. And if you release unsafe models, you will be liable. No one is trying to No one has absolved you of that liability. We certainly didn't. So you know this this idea that like you know if we don't create a federal department of AI this Orwellian you know ministry of truth or surveillance then in the absence of that it's going to be anything goes it's not true we actually
25:08have a lot of laws and then you know like I said we introduced the white house accord on super intelligence where we introduced a framework um again inspired by Sarbain Oxley in many ways where uh the companies the frontier labs agree to create internal controls to create an internal team to enforce those controls.
25:28That team would then be uh those controls that team would then be audited and validated by an external auditor, a professional who goes in and does that and then they report to a committee of the board. So that was an approach that we thought was something that could work well and again that could be the basis for a legitimate um legitimate legislation if you want to go ahead now and make that permanent. Um I think the idea was to kind of give this a chance
25:57to work before you pour concrete on it. But just make sure we get it dialed in. But the administration now is you know rolling with that framework and uh making sure it gets implemented.
26:08Okay. I you talked a lot about open a hugging face the situation there.
26:12Um tell us what that actually is. Help us unpack it and what's the importance of open choice of open source models in that context.
26:20Well, the um the the hugging face incident, it's very famous incident now where some models or a swarm of agents uh that were uh in the the lab so to speak at OpenAI. I think they were probably doing training um managed to break out of this the sandbox was misconfigured and the agents got out and they got off the internet and I think they were running a test.
26:48They were uh testing it on uh on exploit gym which is a a cyber benchmark and the um the the model that they were testing had the idea that the answer key might be um at hugging face.
27:04They wanted to cheat and so they you know the the the agents um decided to see if they could break into hugging face and get the answer key. Um and this is this is a problem in model training is that you know is that if you put tremendous amount of pressure on an agent or a model to accomplish you know a a goal um but it doesn't have the right you know hasn't been trained on
27:33the like let's say the right ethics then it could decide to solve that problem or it could decide to go cheat and find the answer key right and so that is that is a legit training problem that's going to have to be addressed. Um, in any event, uh, you know, there's other things there other lessons from that as well. I mean, clearly you need to have continuous monitoring of agents. The, um, this whole episode was going on for many days, maybe even weeks before anyone even noticed.
28:02So, that was a huge fail. Then the sandbox, like I said, was misconfigured, so it was easy for the agents to escape.
28:08Uh, I don't know that it was on a separate network. So there's all these like layers of containment that were never properly put in place. And I think that's a big lesson here is that you can't just rely on the model being aligned, you know, trained on the right, you know, values. You also have to have continuous monitoring and containment of these powerful technologies. And so that's all being developed. But I think one of the great ironies was that when Hugging Face saw that it was being
28:37broken into, that it had a had a you know cyber attack on its hand, it tried to use OpenAI's leading model to analyze what was going on.
28:48And it they were turned down, right? because the guard rails that had been put on chat GPT couldn't tell if Hugging Face was trying to defend itself or engage in a offensive cyber attack itself. So, it just said no. And as a result of that, Hugging Face had to turn to GLM 5.2, which is a Chinese model, right, for cyber defense, right?
29:12And to me, the takeaway from this is that a fully centralized approach to AI safety is not going to work. So you know by centralized I mean that you just you know you you have gatekeeping or guardrailing of the capabilities. So you just you know you have you introduce models and then you just say that there's certain capabilities that people can't have. The problem is the bad guys always find a way to get the capabilities. And the reality is that there's enough open- source models
29:41coming from China or elsewhere that the bad guys are always going to find a way to get the capabilities. And so we have to figure out a way to make sure that the good guys have all those capabilities too. And I'm not saying that, you know, we shouldn't have certain guard rails, but you know, it's it's a very tricky balance because if all you try to do is centralize and limit who gets what, again, the bad guys figure out a way to get the capabilities and then the good guys don't have the
30:10capabilities and then they they have no way to defend themselves, right?
30:13And that's that's a serious problem. But anyway, it just shows like what a nuance problem this is and um you know obviously cyber is a big a very big topic right now and uh you know we we just have to have decentralized as well as centralized um solutions to this problem.
30:28Okay, great man. So last question here um on that note where are we on US open source and what do you think America needs to do to win the IRA?
30:38So you know as we've been talking about open source is very important. Um I think that you know in the larger in the larger race I do think that the US is ahead at every layer of the stack. Um but when it comes to the model layer this one area that you could say China is definitely ahead is has been in open source and a lot of that just has to do with how the market evolved. It is sort of a a quirk. It's not what you'd expect. We have a more open system than
31:07they do. you'd expect we'd be the leader in open and they're more closed so maybe they'd be closed but in any event this is kind of the way it's it's worked out and I think that um you know I think it has to do with with business models I mean anthropic and open AAI were able to make their business model really work because it's closed they sell access their API on a on a meter basis has been very successful the um the Chinese companies
31:36are uh went open I think partly as a strategy for catching up because going open gets all the non-aligned developers on your side. So I think that's why they went open and so they and they've been running with that and I think the question in the US is how do we create the right business model for open and there's definitely people doing that.
31:56Um, I would say that I hope Nvidia gets back on its feet very soon because Nvidia was the leading champion in the US ecosystem for open, right?
32:08And they were the greatest sponsor of it and they had the biggest you could say business incentive for open because they want they want as uh they want the market to be as diverse as possible, right? I mean, if you're the leading chip company or any chip company, you don't want there to just be two buyers, right? You don't want it to be a monopsin of just there's open AI and anthropic and that's it, right? So they were trying to create a diverse pool of potential buyers for their chips which
32:37part of I think is is very good and so they were sponsoring the open source ecosystem in the US in a big way. And I hope that whatever happens with that company we don't lose that because we really do need that support if we're going to catch up in this one area that I think is very important. Right. So, you're going to join TK at the protest to bring Jensen back. [laughter] What What TK TK said he's going to get a bullhorn um that's uh coated the chips and lead the
33:05protest to bring uh bring Jess back outside the office.
33:08Well, TK said something else which I which I thought is really interesting. He said um you know the the the greatest fear he has is having an AI overlord.
33:16Yeah. And I that that that is that in a nutshell does summarize look I I'm not saying there aren't safety issues with respect to super intelligence but of all of them that's the one that I worry the most about is you know is that we end up with one or maybe two a monopoly or duopoly situation which we've seen in lots of other tech categories you can almost say it's the norm right and then that company works handinand glove with the government and you know that you
33:45basically you have a concentration of power in very few hands and they determine who gets what capabilities, right? And that that is a dystopian future. That is an AI overlord. And I do think that, you know, everyone who's paying attention to this should try and make sure that that doesn't happen.
34:03Let's not do that. Yeah. Awesome, man.
34:05Thank you. All right. Thank you. Great. [music]