0:01I think a lot of the cyber issues that we've seen in the last six months or so could have been addressed with better engineering, more robust engineering.
0:09The next phase which I think carries the most risk is closed RSI, closed loop without the human reviewer process and that will obviously be faster but I think is also a very dangerous way to get to full AGI and our international institutions are very very weak as we all know just at the time when we need them to be strongest in in a way. So that's a bit unfortunate. But by the way, British a bit unfortunate is holy. Just for translate to translate another translation. Yeah. It's like tea time and dinner time. Exactly. A bit
0:41Please welcome Dennis Hades and Reed Hoffman.
0:49Dennis, thank you for joining us.
0:51Great to be here. Right.
0:52Thanks for having me.
0:52We've known each other for I think well over 10 years now. maybe approaching 12 or something like that. And uh it was an honor when you asked me to be on the AI ethics board for Google for you know for our various funny meetings. Uh yes and it's been great. So one of the things we've been trying to do uh at the summit is to be envisioning the world that we as we'd want to make it. The theme has been agency and you know as you have been a leader in and
1:22knew well before most people were thinking about AI that one of the key things for thinking about AI for human benefit is medicine and medical. say a little bit about how you envision this, how this helps, you know, people say, "Oh, if I could be getting that, I understand why you AI guys are actually trying to build something that creates a longer human experience, a more quality of experience, something that's not it's you call it anti-exrisk."
1:49Yeah, for sure. I mean if we if we take a step back for a moment is like um big picture and and you know this having been sort of um knowing us from quite early in our deep mind journey that the the whole point of me working on AI for my whole career was to use it as a kind of like the ultimate tool to to advance science and and medicine and our understanding of the world around us but also improving human health. I've always said that the number one thing we should use these amazing tools that we've now built and we were, you know, dreaming
2:19about 20 years ago is to improve the human condition and improve human health. And I've tried to do that certainly with my own uh kind of more personal projects. uh obviously things like Alphafold um but also uh the the spinout drug discovery company Isomorphic um I've tried to push it really hard in that direction and as I know you have too more recently and I think it's exactly the sorts of things we need to be showing the world that AI can do. I think um you know what better
2:48thing than to you know use AI to help us cure terrible diseases. So some of the protein folding was this kind of intersection of the idea of like we have really good data. We could use scale learning because the whole learning like the the cognitive revolution from learning starting with even pong but the you know and selfplay and all the rest is like okay that's this would be a really good way of doing it. How does um like the kind of the alpha fold lead to medicines? Like what's part of the arc and and you know because one of the
3:16things is a question I get frequently is like yeah yeah you're talking about that but is it true? Is it going to happen anytime soon? You know D. So I'm you know I'm I'm passing along the question I get to you.
3:26Yes. Sure. Well look biology and um and is the most complex you know systems in the universe right? So that's why it's so hard because of the emergent behavior um the interactions and proteins are the building blocks of life. That's how um I've always seen them and and the protein folding problem which is what we you know tried to solve with alphafold was considered to be like a 50-year kind of grand challenge in biology. It's sometimes described to me as as like
3:56Fermat's last theorem but for biology and it's critical to understand the structure of a protein from its genetic sequence it amino acid sequence to understand its function and that's really important to understand basic biological mechanisms but also to understand um you know what goes wrong in disease and what things to target you know if you imagine you're designing a drug then you need to know what the 3D surface of the protein looks like so you know which part of that protein uh to to
4:24bind into. So it was always a critical but not obviously only still a small part of the entire drug discovery process but it's absolutely critical part of that and we were able to you know with AlphaFold accelerate it you know by by orders of magnitude um um by building um AlphaFold and if you think about it we finished AlphaFold 2 back in 2020 which is you know six plus years ago now and it's still probably the best
4:50example of concrete example of AI I applied to a really hard scientific problem uh that was thought to be sort of uncrackable. Um and so that just shows you how far ahead of its time it was. And now you know we're really proud of of the impact it's made. Like 4 million researchers around the world use it. Pretty much every biology and medical researcher in the world almost every drug in future will have made use of alpha fold all the structures. But it's not enough. We want to expand it to biochemistry and um also predicting
5:20things like toxicity or adne properties, all of these other things that you need to know about before you can have a drug that's safe to use and that's efficacus, you know, efficient in the in in whatever it is you're trying to tackle.
5:37Um, so speaking of accelerations, uh, let's go to one of the hot topics of the day, uh, recursive self-improvement.
5:46Uh, what is your kind of theory of the game? And I say this because you, you know, are a master games person. So I use that as kind of deliberate. And if you don't know what I'm referring to, either look at the documentary or look it up on Wikipedia. Uh, so like RSI, opportunities, concerns.
6:04Yeah. Well, look, RSI uh it's obviously an extremely uh useful technique to make progress. It mostly works best in what I call verifiable domains, what's called verifiable viable domains. So, um math is one, which is why we're seeing so much advance there. Coding's another um but originally it was games. Actually, one of the first sort of famous recursive self-improving systems was one we invented quite a lot of years ago now called Alpha Zero, which was off the
6:34back built off the back of Alph Go, which was the kind of, you know, famous program that beat the world champion at Go back in 20, you know, 10 years ago now, 2016, and kind of was the beginning, I think, of the modern AI era. And then we built a second version that was in some ways more consequential, Alpha Zero, that didn't need to use any human data anymore. And um it just basically played against itself. Started at random and then you know around a dozen to two dozen different epochs. You know it play like
7:02a million games against itself. Uh it would then train a new version of itself uh uh that was slightly better than the old version and then that would play a million games against itself and so on and and for example and then it could play any game not just go it could play any two-player perfect information game.
7:19And for me it was especially visceral as a chess player because I remember watching it play chess and within um you know 12 hours basically or less than 12 hours it would start from random play in the morning to you know you'd have your coffee break at 11:00 a.m. and I could still just about beat it. Then by lunchtime it was grandmaster level and then by by tea time it was better than um the any any chess playing entity.
7:46time is dinner time for the US computer time to you know or computer you know kind of handcrafted chess computer is better than all of them so we sort of saw the power of that recursive self-improvement and obviously the thing that all these domains have in common is you can verify the answer is it cor is it a correct answer is it an optimal answer is the equation correct it does the code run it did you win the game and the important thing about a verifiable domain is you can verify the answer um so you know when you've you've you
8:14you've you've hit the by point. Um, but also that means you can create almost infinite synthetic data because you can just use the current best versions you have to set new problems or make new moves and then um uh just sort of verify uh uh whether the synthetic data was correct. The problem with synthetic data a lot of the time is you can't verify that it's right um from the right distribution but in these cases you can.
8:39So we've seen the enormous progress with maths and with um coding and coding's obviously become enormously valuable and everyone is having to do that now. But I do worry about it because in the limit so there's different forms of RSI right now. We're in the the early days of RSI I would say where what's happening is and you're probably all experiencing this as well with your teams is um you know it's making a human engineer maybe three, four, five times more productive.
9:09Okay. So, uh but the but the human is very much still in the loop and reviewing code and architecting things and um and and and writing code still.
9:18Um but it's all just enhanced. Um the next phase which I think carries the most risk is closed RSI closed loop where you basically manage to somehow figure out how to do it without the human reviewer process uh in this in the in the whole procedure and that um will obviously be faster but I think is also a very dangerous way to get to full AGI because we're likely for two reasons it's it's sometimes used to be referred to 20 years ago as like an intelligence
9:45explosion because you don't quite know uh how you know how do you control keep control of such a system uh such a process and then the end result will likely be quite alien. It's going to be like an alien intelligence. There's no reason you to think it would be sort of in the similar kind of mind space that uh humans are in. So that will make it very hard for it to understand for us to understand it and maybe for it to understand us. So I think it's you know we should be very cautious as we get
10:14closer towards that closed loop form of RSI and I think all the AI researchers sort of know that but there is a sort of race to the there's this sort of intense obviously uh uh uh competitive race to improve your coding systems. So presume that the the incentives to be in a race continue and you know you and I some people really know this you know have tried to figure out if there's various ways you can get global collaboration to you've dedicated a huge amount of energy
10:42to that so thank you um uh it's a hard game as you know but presume that the competitive pressure for continuing to do RSI and coding continues in that environment what are the best ways that we can potentially navigate that because we have to see kind of where RSI is going, what it's pointed at. What are the kinds of things that the, you know, the leading labs such as your own and the others, what should they be doing to try to navigate this environment? Well, what we're doing and I think other leading
11:12labs are trying to do is you got to uh so there's there's more prosaic things like make sure uh uh cyber defense is accelerating faster than cyber attack and making sure you give access to trusted cyber defenders first on on maybe slightly fine-tuned versions of your general models that are fine-tuned for cyber defense. Um and so you keep the defenders ahead of the attackers.
11:35It's a pretty tricky problem to do in general. um also building much more hardened sandboxes. I think a lot of the uh cyber kind of issues that we've seen in the last six months or so could have been addressed with better engineering, more robust engineering. I think better you need really amazing monitoring systems. You know, they shouldn't be possible for agents to be running for for weeks without anyone realizing. You know, these kinds of things that we've seen some of our uh some of our competitors and others maybe, you know, it's just the beginning of this
12:05happening. So I think everyone will get rapidly better but they have to as well because otherwise something bad's going to happen inadvertently. Some kind of incident you know I sometimes call it like an oil spill is going to happen on the current trajectory if we're not if we don't get uh collectively if if everyone doesn't pay a lot more attention and put a lot more resources on these types of systems eventually you kind of want formal guarantees maybe some bounds around what these systems can do uh test how robust like the things like the guardrails are. This is
12:34actually something I encourage a lot of top mathematicians that I know well who are maybe, you know, pretty sad about the state of pure maths, but they their skills would be incredible uh if they pivoted to these types of domains, more like complexity theory, game theory. How do you uh and of course when they're going to have to start dealing with swarms of these agents, potentially collaborating in some way. Um how does one keep track of all of that? And should there be ways that the labs are
13:01encouraged, required, permitted to collaborate on this? And if so, what should those what should some of those ways be? Because it obviously gets stronger if there is sharing of intelligence, methodology. Yeah.
13:14You know, mutual accountability.
13:16Yeah. I mean, there's there's been lots of proposals and I've been working on these things a lot. And you you you know you saw my essay over the summer, maybe some of you read it about potential for like an SRO industry run standards body with some kind of government uh oversight over it. Um I think something like that is going to be needed and then ideally it becomes international. You know it's USled and then obviously the state of geopolitics today makes it and and our international institutions are very very weak as we all know uh just at
13:44the time when we need them to be strongest in in a way. So that's a bit unfortunate, but we got to start somewhere.
13:50By the way, British a bit unfortunate is holy. Just for translation. Yeah. It's like tea time and dinner time. Yes. Exactly. A bit unfortunate.
14:01and um and and so we're trying to you know various of us are trying to do things about that and and that essay did have an impact and uh you know in the white house and in administration and and with our colleagues and eventually you know I think some of that energy became the accord that you the famous letter everyone signed which is which is better than nothing. I don't mean goes far enough but it's a start and at least it acknowledges from all of the main players uh uh the issues that are there and and that they need to be addressed.
14:30doesn't say too much about what to do about them but you know that has to come next and at least it gives air cover I think for the teams to work together on you know what sorts of benchmarks what are safe best practices on safety what are you seeing you know uh uh in terms of analysis of what things could happen with these types of systems and I do think there are some things that generally all the leaning labs do agree on like um the idea of third party auditors or evaluators coming in obviously there's a question mark which third parties and and I think we need a
15:00healthy kind of ecosystem of that that actually you know you could have good startups and companies doing that that that that there's enough revenue there for them and there should be given the amount of investment going into AI as a whole bit like there are auditors you know ini in the finance industry or any of these industries so I could see that happening um and some of the big players are moving into this like Accenture we're working in other leaders are working with I think they're a little bit ahead of the game because they've acquired you know a good startup good friend of ours Mark Warner startup so
15:28they have some expertise internally but I think we need more of that and then other things like you know the leading labs maybe red teaming each other uh before uh general release of these systems so let's move from that to a little bit more of the fantastical you know it started actually a few years back I think there was a Google engineer who said I asked Gemini if it was conscious and it said yes so therefore it's conscious which seemed like a little bit of a a weak inference that's now gaining some motion with some of the you know
15:57things especially anthropic is doing and and in and the way that it u kind of strategizes thinking that well at some point it's almost like a Pascal's wager at some point there will be consciousness in these systems and we want want the those conscious systems to look back saying that we were treating them the way the same way that we like treat a 2-year-old um which is a hey look we were good to you as a 2-year-old and so you know it kind of continues through obviously to many people that looks kind
16:23of wacky right and um and so what do you think I mean cuz Part of what I think maybe many people don't realize is you also come to this from a depth of expertise in brain science and you know one of the things is not just a oh did the chatbot say it was conscious or not is not the way to make a decision on this kind of thing.
16:41What do you think some of the ways to think about this is and also how we should be deliberately trying to shape it.
16:46Yeah. Well look I think there's quite a few things to say about this. First of all um we haven't despite thousands of years of thinking about this there isn't a agreed upon definition of consciousness. So that's one one issue I think and that's one issue with labeling things conscious um especially the hard problem of consciousness you know is it really feeling this internally like the way we we seem to you know behaviorally these things I don't know about you but yes exactly but the behaviorally well but I used to argue with Daniel Dennett about this that there's an additional thing is we're not just behaving in ways
17:16like we'd imagine conscious beings behave but we also share the same substrate we have the same squishy you know brains and so then it's most parsimmonious to think you're feeling what I'm feeling But of course with a with an AI that's not the case. You don't have the second the substrate equivalent. So I think it's going to be very hard to conclude that. I think there are dangers. Look there's there's two reasons why I think we should be focusing on building intelligent tools today not uh conscious entities. Um I think for one thing is a safety argument
17:44and uh technical uh safety and risk which is that you know there's a direct trade-off between creating something let's say uh with selfition right or self-awareness um you know even prior to consciousness these are the prerequisites right to a conscious entity um that it will and that and instruction following or sometimes called courageability. So this idea of a system actually following instructions and if it's going wrong you telling it
18:13you know you as the user or the the human user telling it like stop doing that right or switch yourself off or whatever it is that you're going to you know change change course. So if you are training your system to be able to reject commands based on its own internal uh workings, then you're setting up a situation where if you've got your constitution wrong or your rules wrong, and this is what Asimoff was always warning about with these three laws, you know, that was a very simple form of a constitution. And his
18:43books are all warnings that that's you're never going to get that right in every corner case, in every edge case.
18:49And and and and I think his books show that very well. And yet we're sort of I think a bit hubristically thinking that that some people thinking that you know sometimes called the misalignment problem is sort of under control just because today's systems are you know the good news is they seem more malleable than we might have imagined 15 20 years ago which is great news in terms of their steerability but that doesn't mean we've solved um the misalignment problem in total as we get closer to you know a GI and super intelligence and so on. And
19:18I just think um it's it's it's dangerous to think one has. So that's the first thing. There's a technical problem especially if you combine it with something like an RSI.
19:28Right. So RSI with self self valition is you know sort of doubly dangerous. So that's one thing. Second thing is a societal more of a societal issue which is that I don't think we are ready yet as a society. We haven't had the debate with a wider society, philosophers, theologians, um, artists, humanities, not just technologists as to, you know, and from many different cultures. Like, do we want to cross that second rubicon?
19:55Are we ready to have the discussion about do these systems have moral standing? Um, do we need to worry about, you know, what are we doing with them in this box? You know, are they slaves of some sort? you know, all these questions come up that are complex questions and I don't think we're equipped to even begin to start dealing with those questions.
20:14So, my view is we should um cross the first Rubicon uh on its own which is build these intelligent tools and then actually use these intelligent tools to do more neuroscience to go back I'd go back and do more neuroscience to actually understand what the neural correlates of consciousness really are.
20:31finally understand that and then you know that'll probably take 5 10 years and then cross the second Rubicon well to have a discussion about should we cross the second Rubicon and then if so how right of of the of the conscious entity type and how would you cuz you know you engaged in hours of these discussions how would you answer the folks who say look the only way we get the intelligent coralate to be powerful enough is it it has as it were the ephemeral accidental it actually has this you know cognitive state and so
20:59right so we should talk about this a We discussed this you know even 10 plus years ago and I discussed it in my PhD back 20 plus years ago of look it I felt and I still feel intelligence and consciousness are double dissociable which means that you can have an intelligence system that's very smart on IQ and so on kind of like the systems we have today which I don't I personally don't feel are at all conscious yet right from my own you know everyone probably has their own view but I don't feel yet and then you can have conscious
21:27entities uh that are not that smart for example my pet dog. He's lovely, but he's and he's pretty conscious. I think he dreams and everything, but he's not that smart, right?
21:36And that's provable.
21:37So, I think you can have double dissociation here. You know, many of the higher animals, dolphins, monkeys, and so on, elephants, I think most of us would view their conscious beings, but and they exhibit all of that that sort of behavior. And of of course, they're also made of carbons and and the same neurons and types of things we're made of. So, I feel like it's pretty obvious they have some of those qualities. um but they're not necessarily that smart.
22:00So it feels like you can dissociate these things and and I think to the extent that you can I feel like we should you know we're coming up close to the end of our session and the end of you know the summit sad um what should we think about in terms of the whole notion of AI sovereignty right because obviously one of the one of the things that is kind of going around in a lot of countries right now is oh is this really important and you know one of our earliest conversations was with you trying to solve this
22:29problem so I wanted have share some thinking and obviously it's updated for whatever that was 13 14 years ago but well look I think first of all first thing to say is it looks to me like there's going to be you know half a dozen you know six to eight if you include Chinese companies company you know offerings at the frontier near the frontier and it feels like there's going to be leaprogging and maybe something's like three to six months ahead but it's going to be uh relatively close so there's a lot of options if depends if you're talking about an enterprise or a
22:58country right So I think as an enterprise you have a lot of options to arbitrage between these different things. Um some of them are better at certain things you know multimodal understanding versus coding or it's very hard to be better on all vectors of importance at once. So I think that's going to happen and then open source um which also has its risks because how do you as we've discussed for many many a year is how do you restrict um bad actor using that whilst enabling of course as many good actors scientists and others
23:27uh to use uh these fantastic tools. So that's an unresolved problem but open source is you know maybe only lagging 6 months to a year behind. Uh obviously we need kind of western open source solutions. Um but I think at this point if we're talking about countries as well all companies that need their data to be private or they want to fine-tune further on their own you know sensitive data again be that enterprise data or or sovereign data. I think that's that's going to be one solution. There's also you know looking at what are the
23:56downstream things that can be done with these tools you know big companies and big ve sectors that can benefit from these tools like healthcare that you can just use this you know almost like commodities and then build the value on top of it rather than you know I wouldn't recommend every country for example trying to build a a frontier model that just doesn't make sense to me so we're going to they're going to have to find some other way um of shortcutting that well and also collaborating and uh working about arrangements with
24:24multinational companies like your own. I think one of the reasons I wanted to kind of end on that question is that part of the theme of of the summit is agency and agency is not just a question of individual agency like how do you take agency but it's also a collective how do we take agency and I think part of the thing that's actually important in the AI age is that we take agency not just as any one country but as countries together it's part of how we have a we are in this together and we are we are building for
24:53the benefit of humanity and I think that's the kind arc of we started with Sarah Frier and thank you for joining us because Demis has been at this with such intensity that I was so glad grateful that you set up the agenda for me thinking about how to help in this from our very first you know discussion at King's Cross. So uh let's give Demis a hand. Thank you. Thank you.
