0:07All right. Well, welcome to the inaugural YC debate. I'm Vivian Meden, uh GP here at YC, and I'll be our moderator tonight. Uh the topic of tonight's debate is someone on it's something on everyone's mind, I'm sure.
0:21Uh AI is already extremely powerful and improving very quickly, and so should governments slow it down? Specifically, the resolution tonight is governments should intervene to substantially slow the development of frontier AI. So, tonight we have here Jacob Coxin who garnered national attention after leaving a research role at Enthropic over concerns about the dangers of AI takeoff and he will be arguing that the government governments should
0:50substantially intervene. And on the opposing side, we have Robin Hansen, an economist at George Mason University, who will be arguing that they should not. Robin famously debated Eliser Yudkowski about the speed of AI takeoff 15 years ago in a debate that remains a touchstone uh in the AI community. So, a couple logistics for tonight. In a few minutes, I'm going to invite Jacob and Robin on stage. They're going to give seven minute opening debate statements
1:19beginning with Jacob and then going to Robin followed by five minute rebuttals and they won't be allowed to interrupt each other during those times. And then they'll do about 15 to 20 minutes of direct exchange. We'll see how it goes uh where they can go back and forth and then we'll do some audience QA so folks in the audience can uh have their concerns directly addressed. And then finally, we'll hear the hear the closing statements and do a vote on who you think won the debate. And that's going
1:47to be on the merits of their arguments. And before we get started in our timehonored tradition, we are going to do a initial vote on the resolution. So you can make noise in whatever format you'd like. Stomping, clapping, yelling, whatever you'd like. And so uh first let's start with the affirmative that governments should slow down AI development substantially.
2:12[applause] All right. And now for the negative. Governments should not intervene substantially to slow down AI development. [applause] All right. So remember those noises, folks. And now, please join me in welcoming Jacob and Robin Hansen to the stage.
2:35[applause] [applause] Cool. I'll start my opening statement, I guess. So, my argument has three components. The first is we're on track for super intelligent AI systems, possibly within the next two years, in fact, probably. So, broadly, super humanly capable. The the second component is that we don't yet know how to control their drives, what they do.
3:00And the third and perhaps maybe most important component is that at the current pace companies are not incentivized to address these risks.
3:08Seriously, they'll require outside intervention to address these risks. So to move to the um the capabilities bit, I think actually with a lot of um people that are not scared, I think they just don't imagine the same capability trajectory as me. So I think they're less concerned about the the capabilities. And I'd like you to imagine a broadly superhuman intelligence, something that is, you know, better at podcasting or speaking or any sort of creative act as well as the technical stuff, the coding that we that we're used to. And there's a lot of
3:38evidence that we'll see this probably by the end of 2028. And the argument here roots through 2027. So in 2027, I expect to see a superhuman AI researcher. Um, ev evidence for this, there's a lot of evidence. One is just lab people are saying it. A lot of them like no brown who I think you should trust more than me says that the the generation after the next will be able to do his job roughly. There's also uh outside view trends. So you can look at the fraction of AI research and development that's that's happening using claude and it's
4:07currently at 25% anthropic. If you extrapolate the trend bam late 2027 all the research is is clawed if you follow the trend. You can also look at the trend of self-reported speed up. So humans can report how much they're getting a speed up from AI. It's currently around four to 5x reported at anthropic. That's maybe an underestimate. The the report is maybe untrustworthy, but it's doubling every 3 months roughly. If you extrapolate that forward, you get a 100x
4:352027, which is sounds a bit like an automated AI researcher. If you have a 100x speed up, you can also look at taste evaluations. Anyway, there's any number of outside things and they all seem to point to 2027 being the year that the job of an algorithmic researcher, the job of say someone at Anthropic or OpenAI will will largely be automated. And I think this is roughly consensus around labs with a date varying between maybe mid27 and sometime
5:03in 2028. And then it hits everything else, which I suspect is one place I I I'm going to disagree with Robin on. I think that once you have the superhuman AI researcher, things go very fast.
5:14There are a lot of lowhanging fruit left in in our general intelligence algorithmic improvements. Like we keep seeing new discoveries that make models broadly more intelligent. Once this is automated, we could compress a lot of essentially AI software research into a short period of time. Imagine say a decade happening between mid 2027 and mid 2028. And we know that improvements to general intelligence transfer to being more data efficient at learning new tasks. So if you get broadly
5:44smarter, which the models will, they'll make themselves broadly smarter, you need less and less data to get good at anything else. So currently, say you want to teach a biology, maybe you need a lot of data. A very smart thing will need not much data to to get a lot better. And I think sometimes people are like, well, it'll get the verifiable stuff, it will get math, it will get coding. It it's going to come for everything. Like think about say GPT40 being good at persuasion. You know, perhaps that's because they did some RL around trying to make it more addictive.
6:12That was a weak model. You know, a strong model could be very very persuasive, which is a soft skill that people don't usually associate with these with these AIs. Anyway, I can I can come back to this, but uh in in generality um once general intelligence improvements are automated, I think that you're not you're not thinking about it properly if you imagine having any comparative advantage. Like people say, I'll be running my company with my AI agents in
6:422028. The best way to run the company will be to ask your agents to do whatever you're doing. If you're a if you're a software company that that's what it will consist of. The job of being say a CEO will be entirely automated. And I think this is this should be scary even without all the alignment problems.
7:02I think just this concept and these timelines should be scary. They should make you think about what you're doing and what things are going to look like. But we actually have evidence that they're behaving in ways that we don't fully understand. You know, we have the whole hugging face thing. I'm sure you've heard about it a million times.
7:18There's also consistent disclosures from OpenAI around things like disabling monitors, cheating on tasks. Models can actually kind of control their own chain of thought. Sometimes they they often behave in edge cases in ways that we would not have foreseen, which is deeply problematic if they will be essentially alien minds, you know, like minds with with drives, smart minds. I think we we want to know how they function. And I think Paul Cristiano had a great analogy
7:48of thinking about the hugging face incident where they tried to trick the graders or they thought a lot about graders and imagining that hugging face incident happening, but they were smart enough to think about humans. These these AIs were trying to get a big score and in service of that they they hacked, they tricked, they tried to edit their logs. They didn't think about humans, but humans are a factor in assigning the score. You can imagine a smarter model would start to think about human oversight, crippling human oversight in order to attain its goals. And it's very scary to think about. Uh and finally, um
8:17the incentives are not sufficient to restrain companies. It sounds crazy, but they are going to race into this. This is their explicit plan is to build these smart models to race into super intelligence without a good plan for how to control it. And that's kind of crazy. Um but they're scared of each other.
8:35They're all worried the other one is less safe. They they need someone to step in and solve the collective action problem. And one more example of their craziness is the security measures are ridiculous for something that's essentially a technology of national security importance. If you want more evidence that they're behaving in crazy ways, look at the level of security they're applying to something that is the most important national defense technology. So in summary, I'd like you to envision how scary a broad super
9:05intelligent mind would be. I'd like you to think that about the arguments that suggest we will have such a thing in 2028. So soon in two years and then you know think about how scared that should make you.
9:18I hope we can have more of a conversation than the format doesn't get in the way. Uh but I'm happy to talk about this. I've been near an adjacent communities like this for many decades now and my one factor theory of what we have in common is a taste for abstraction. We like abstractions. We like to reason abstractly and we have great fun doing that. But there's a danger. Abstractions drop detail and the detail often matters and we like to leap
9:47ahead with our abstractions and forget that we've dropped a lot of detail. So here we're using this abstraction of a super intelligence of the concept of intelligence as this almost one-dimensional factor varying different kinds of machines or systems.
10:05And in economic history, we've seen that this simplistic model of technology has consistently led people astray in predictions because when new technologies like the steam engine or electricity or the personal computer came out, people uh expected much faster adoption and impact of those technologies than actually happened. And our standard understanding is that in order to get the advantages of electricity or whatever it was, we had to restructure lots of other parts of
10:34societies to take advantage of that which delayed and slowed down the impact of those technologies of typically decades after the initial application.
10:44That is the world is complicated and has many different relevant dimensions. So I think even if Claude is able to write lots of code for anthropic that's one small set of dimensions in a large economy what I mostly care about is the impact on the large economy and they won't suddenly get half of GDP because they can make their compiler run faster. So I question
11:13this one-dimensional model of intelligence as there's just this one thing and if anthropic can just study itself for a long time, it rises on this one number and then everything's over.
11:22That's this one parameter model of intelligence that I think is just wrong in the sense that we've understood the economy for a long time understanding there are lots of different tasks and people have to not just be generally intelligent but we have to spend a long time learning to do many different tasks in the context of that book learning isn't usually very useful you typically have to be in the context of a task for a while to learn how to do it and AI will have to do that too just because
11:52it's smart doesn't mean it suddenly can do everything and that means there'll be a long delay of application of AI into the economy which I think is fine usually in history we adapt that is foresight is rare fortunately because it's hard we mostly adapt to problems i.e. We watch for things to go wrong and then we change what we do. And AI is one of the most watched technologies there is in terms of these companies are
12:21absolutely watching how they're used and watching for problems and trying to adapt to it. And that'll go better than typically having foresight when you can't see very well into the future. So uh I would be happy to just see slow gradual adoption where we watch how things are going and see how things change and react to those things and we will be able to react to those things if change isn't crazy fast. Uh so the question is will change be crazy fast?
12:50Now first of all I'm going to say on our priors like the scenario being painted is just way out of the distribution of anything we've ever seen before. So you should on base rates be suspicious. You should be suspicious that it's a abstract argument that's dropping lots of details that's not paying attention to the complexities that usually happens for a person to learn a job or for the the the AI systems have been getting traction mostly in these areas where they have verifiable data i.e. coding and math and they haven't been getting
13:19that much traction in other areas of the economy. It's a slow adoption process.
13:23I'm hopeful that eventually it will but that's going to take a while. for the AIS to learn how to do other tasks and to actually integrate in the economy to do them. But my highest level point would be I don't want policy to depend on disagreements about parameters like this. I want robust policy. So I don't know if this means I'm disagreeing I'm agreeing with the thesis or not, but I'm much more okay with using liability here. So Daario's uh you know thing a
13:51few weeks ago about pacing the frontiers he pointed to a scenario of a hundred billion dollar damage in the internet sometime in the next year which does seem exaggerated but I'd be okay with requiring some providers to have that level of liability insurance and now you see if in fact insurers look at that and go we can't cover that you guys are way too risky they will demand too high a price for the insurance and then they will just have to stop doing a lot of things until they can convince insurers
14:21to that they are careful enough and that would be the right policy for the situation where in fact it's very dangerous. On the other hand, if the insurance companies go, "Oh, sure, fine.
14:31We will ensure this at this low cost cuz we don't see this as a big danger." Then it won't be much of an impediment and that will be the right thing to do. That is, I like a robust policy that will do the right thing in either case.
14:42Especially, this is important to me because it looks to me like regulation in this area is extremely dangerous and difficult. We mostly regulate things well when we see concrete specific problems like drugs that hurt people or car crashes and we make regulations targeted to those particular scenarios, plane crashes and we focus the regulation and processes on the actual problems that we've seen. But here we're supposed to approve a regulatory body
15:12without those specific problems being seen. A pretty abstract large space of possible things could go wrong. This has to be global regulation. It has to be of all things called AI. But AI is a pretty vague category. And regulation's just gone pretty bad in the past when we've given some power, regulatory or discretionary power to do whatever it wants about a vague problem.
15:37You have to believe that in fact regulators will be captured by some powers or other. That's just a political fact. So you have to ask whoever captures this regulational power to decide who gets to do what AI in the world is that a better world than maybe taking the chance of some pretty speculative things going somewhat wrong.
15:57Thank you. We'll move into the rebuttals.
15:59I think the the first thing I have a big disagreement with is your claim that there's no such thing as a single number or the general intelligence can't really be um compressed. And I think that the lesson of the last six years has been that the compression the language compression capabilities of an AI system reliably predicts some notion of general intelligence. And I think you can feel it with say user intent. Suppose you're playing with Astra and it has this subtle feeling for what you want next.
16:27It's kind of a fuzzy generality. This wasn't there a couple of years ago. This this is an emergent property of improving general intelligence. And now any any skill that we have data for we can get better at and the rate at which we get better is determined by the general intelligence. This is just what we've seen. So for example in math we see progress in the future in things like forecasting where it's verifiable you can try and predict things we will see progress in domains like games with all their strategic implications say
16:56strategy game playing if it's verifiable we will see progress and it will be proportional to the the general intelligence. Yeah, I think if you've been watching and feeling the capabilities, you can see that in every area. You know, you've seen this music videos, it's writing music videos in Python, and this is mostly just emergent from being a smarter base model. And we're about to see that completely take off that that ineffable like the general intelligence. Uh in answer to your other other points, um adoption, I think, um
17:24adoption is going to be made easier by the fact that the thing can self-install. Like I don't know if any of you have clawed integrations in stack. It will read your docs. it will figure out how to help for for information work. The models just drop in. It's actually very easy. It's a lot easier to integrate than any other big technological revolution. And from a danger perspective, it doesn't actually need to be integrated to pose a threat.
17:46If the models if the models internally were to pull ahead, they would be getting potentially very smart without anyone having access or integrating them into the the broader economy. Although I do expect them to be integrated. Um and going to liability, I think, yeah. $1 billion, like $10 billion, even hundred billion dollars, you can get like liability for this. It sounds insane.
18:07We're talking about 10 trillion, like hundred trillion dollar risks. Uh there's no market for this. It's an unpreced unprecedented thing. The notion of a technology potentially imposing human extinction. So, normal liability just won't cut it. And the base rates, I guess all this does come down to, yeah, this all sounds abstract and far-flung.
18:29If we're just being sensible and normal, then nothing ever happens, right? This is it's going to be the same. There won't be this big crazy super intelligence thing. I do think if you don't feel it already, we'll feel it more and one of us will be vindicated in the next year, the next two years. So, I'd say yeah, if you don't um feel the surprise of the situation, the fact that this is an unprecedented situation, I predict you will feel it pretty soon.
18:55You also mentioned that the regulation will need to be global. Uh and I think that is true and it's an an awkward fact. I think again justified by the fact that this is an unprecedented technology and regulatory capture again a tough question.
19:14There'll be some body that has to decide when when things are safe. There are proposals like say Elon Musk's proposal of having labs adversarily test each other. So having many independent auditing agencies perhaps some of which associated with the labs is an example of being able to do testing in a way that doesn't centralize authority to one bureaucracy. You know many labs are kind of adversarily testing each other. So I think that that there will be ways to work around it and ultimately it's a a choice between difficult but tractable
19:44political problems and a potentially deadly scientific problem. And I think I know which one I'd rather focus on. So there's no solution to this coming soon. We're not talking about a pause or temporary freeze because there's no technology coming down the stream that would be the thing that could trigger relaxing this. I think most everybody agrees this is just a very long-term difficult problem. So if you are empowering this agency, it is for life.
20:15If you're trying to set maybe a threshold of the capability of a system and regulate that, then Moore's law says the scope of your regulation increases rapidly, exponentially, indefinitely. There is no obvious limit. You're not just li limiting, you know, AI chips.
20:31You're limiting all chips because all chips can be used to do this just at slower speeds. All memory can be used to support this. You're talking about global regulation of pretty much all computing power after a while with no particular end in sight. That's what we're talking about here. And I just don't think the world is up to this sort of discretionary regulation. We aren't really capable of having a global organization like this that we could trust. That is, we have to trust
21:00somebody to say which systems are safer or not. But that's not an easy thing to judge or measure. If we were measuring pollution or something, we could at least talk about measuring it. But we're talking about somebody's judgment about which systems are safer or what. And it's even not even clear it's safe about what. So I'm happy to admit that when we're talking about military knowledge or technology that we makes more sense to restrict military technology. So in the past when we had libraries, we didn't shut down libraries because there might be a possibility of some books on
21:30bombs chemistry in them. We said don't have those books in the library.
21:34Similarly, I can imagine regulation that just says don't have certain kinds of knowledge and the AI is accessible to the public because that's a technology about a weapon. That's different than regulating the machines themselves. I I think I should also mention that I am less afraid of AIs that are powerful and not fully under our control because that's what we've long accepted from our children and grandchildren. I would see
22:03AI as a different kind of descendant and I would encourage you to hold it to a similar standard. We've always accepted that eventually our children and grandchildren would be more capable than us and win conflicts with us and change their values. We didn't expect it to change enormously, but sometimes they have. And if AIs similarly become more powerful than us and have somewhat different values, but largely inherit our culture, which they are doing. We are making them in our image, I'm more
22:32okay with AIs being worthy descendants who in fact are ones we can be proud of.
22:39And we shouldn't try to keep our descendants fully under our control forever. That's never a good idea for children or grandchildren. Yes, when children are immature and uh you know not very well behaved, we have to discipline and control them while they are in their immature phase. But once AI reaches a level of maturity where it understands what it's doing and is a coherent about its behavior, I'm much less concerned. I just the idea that it's just going to kill us all seems to me really speculative here. Even if
23:08there is one dimension of intelligence and it happens fast, that doesn't mean this intelligent thing decides to kill us all. That's just a bit crazy.
23:16Fundamentally, um, you know, we have a disagreement here about how plausible is a very extreme scenario we've never seen before. And it's really relying on a very simple abstraction where there's a lot of stuff that isn't in the way of the abstraction. We're just imagining this, you know, recursive self-improvement. Many people in the last few weeks have pointed out that the recent growths do seem to be showing diminishing returns to inputs. that his inputs have grown very fast but the the outputs as a function of inputs have shown diminishing returns and that
23:45suggests that there isn't going to be a fast acceleration with diminishing returns you get slowdowns just like we've seen with pretty much all technologies before so I was an AI researcher long ago and I jumped into the AI world at that time because I heard you know it was all going to be over soon and I better jump in if unless it'll be too late. I bought it. I was dumb. I was 20. uh but in fact that reached a peak and faded just like many other AI technologies have. That would
24:14be my default for the next one. But again, I want to be robust. I think if they are responsible for hundred billion dollars worth of damage insurance, that'll be enough to handle most of the risks they might be taking, the extra risk you might get by empowering this global regulatory agency seems to me not worth the cost because that seems crazy dangerous.
24:35Great. Thank you. We will now move into about 15 minutes of direct exchange.
24:40I I could ask you uh what do you think about AI as descendants?
24:44I think the difference is our descendants don't like we don't have 20,000 clones of them trying to break into third party infrastructure at way faster than human speeds which is a facicious way of saying that these descendants have ridiculous capabilities. It's a a disjoint change.
25:00And um previously our human children it was more like continuous. It was more like a a small change, you know, like a little bit smarter, new technology. This could be a a sudden discontinuity, like jumping forward a century. And I think a handoff like that is not something I'd endorse.
25:12Is it just the rate of change that bothers you? If it happens slower, you wouldn't mind. It's just that it's happening fast.
25:18I think so. If it happens slowly and safely, then I wouldn't mind the But we're assuming the safely, and we're just saying if your descendants get much smarter than you, faster than you anticipated, you don't like that. You want them to go slow and only improve slowly, just like you did. So you don't want your children to, you know, outshine you in terms of how fast they learn stuff.
25:37I want it to be continuous because kind of continuity is how it will be continuous. It'll just be faster, right?
25:42I mean, discontinuous from a perspective of someone who measures things in years, you know, but what if we measure them in days, then continuous?
25:48I think if you took someone from the 1700s and dropped them in the 1850s, they'd be quite unhappy. Whereas, if you like slowly guided them through, they'd maybe enjoy it.
25:57But then it's just that you don't like change being fast. It's not about which changes happen really.
26:02Yeah. I just don't like the change being fast and I think most people wouldn't like the change being fast.
26:06So our world is changing much faster than it used to. Right. Growth and change has accelerated over the thousands of years and centuries and we've had a relatively steady rate of change for roughly the last century. But before that it was much slower and even slower still. Would anybody want to go back to the slower rate? Are we at the perfect rate of change now? So that you know in the past it was too slow and now it's perfect and anything faster would be bad or I think there's a broad range that's good and we're about to throw it directly into the unacceptable territory.
26:34Yeah. Roughly. Okay. I have I have a question which is um okay.
26:37So if we move to something different like um what what are your rough timelines for say broad broad artificial general intelligence or say automating most cognitive labor? Do you have a rough sense of when you expect this?
26:48My guess is that this boom isn't the boom that does it. Y but some boom soon will uh I my median guess for this boom is that the software industry will roughly triple in quantity in size over the next decade. half of AI revenue will be spent you know in that industry and that will be an internet scale boom which will be great but I predict that the even if claude gets good at compiling claude code and doing
27:17speedups in claude learning that won't remake society that is that will just make more people want to buy cheaper stuff for software which is the main traction so you know you think that AI is because they'll smarter they'll be able to spread much more fastly through the economy. And I'm saying it's that spreading rate that's the key concern, not the rate at which it pass IQ tests in some lab. That that doesn't matter very much. What matters is the rate at which society is disrupted. But then why isn't that slow enough for you? If it
27:47takes 10 or 20 years for AIs to spread through the economy, I guess I just have this model that we take data for every job, for example, and then we fine-tune with that data.
27:55that data is currently being aggregated and quite soon in the same way that we are super human at math we're doing every human cognitive task even if you don't expect any generalization and I'm a bit surprised you expect it to be a lot longer than that or do you expect us not to reach general we just don't have data for most jobs actually if people were you know recording their job every day all days for the last 10 years and we had millions of those then those could be more plausibly automated but we have not been doing that most jobs learn to do the job you have to show up on the job,
28:25watch people doing it, have them instruct you, etc. And if AI doesn't have rich data on how people do jobs, it'll have to do that and that's will slow the application.
28:34There are people out there grabbing data. I hear that's kind of popular here. Uh, right.
28:39They're getting more data efficient, I think, as well as they get smarter, they're they require less data to get get good at the job. So I I think we agree that u so Douglas Lennon famous AI researcher long ago made this image for me that stuck in my mind that each piece of knowledge sits at some level of abstraction and there are some pieces of knowledge that sit at the very highest levels of abstraction but most knowledge sits at much more specific lower levels of abstraction. So a general intelligence mostly can know that
29:07highest level of abstraction things. But even when you know that that doesn't tell you all the lower level things. You still have to learn and have experience to learn the more specific things for each particular part of the world. Even if you're really good at the highest level.
29:21I guess we try we could try and talk about like specific jobs. Like for example like making music videos I think is quite clearly on its way out. I don't know if you've seen any of the I'm not sure.
29:30Well, we we have large libraries of news videos, right? That's where you have a lot of data, right? You're you're pointing to, you know, places where we have a lot of data, to the jobs where there's where there's less data.
29:41Most jobs out there, they they aren't producing their job isn't to produce a video, for example, or a picture or a text. Their job is doing something in the world and that job is being done right now. And in order for you to learn that job, you'd have to show up and sit there and watch them and try things and they'd have to tell you to do it different. And AI can't learn that faster if it doesn't have some other source of data to do what people now do to learn the jobs, right?
30:05I guess we'll see. I feel like for I'm mostly talking about cognitive labor and for a lot of these jobs, people can do them remotely. So all that data is is logs and they are now embedding in these companies accessing the data and training AI models on the data. So sort of anything that's purely on a laptop, provided you're not missing something intangible, I suspect will be data foder for next year's super models. Even in software, you know, recent reports about software engineering firms say that even when individual employees can write
30:35software faster, the organizations they're in aren't doing software that much faster because they're slowed down by setting up the task requirements, doing testing, uh proving integration, those all things are still taking a lot of time. is that's just an example of how if you optimize some particular low task and miss the whole organization around it, you know, you don't understand all the other complexity.
31:00There's this old story economists have about the middlemen. Most people in history have thought there's no point to middlemen. Why are they just getting in the way? But they're missing that you have whole structures that need to be there to make things happen and middlemen are the ones who make those structures happen. And that will happen for AI too. Even if it can do the lowest level task well, it has to be part of a larger system and it has to learn to do that well. And of course, we aren't we don't actually have good data on those
31:29larger systems. Typically, that is usually the people running those don't want to have data because they don't want people to see what they're doing. So, that's a problem. A big problem in the world is hypocrisy.
31:40So, when people are willing to admit to what they're doing, it's easier to get data on it and to implement it. But when people don't want to admit to what they're doing, it's harder to get data and then it's harder to automate that because they're in fact trying to hide what they're doing.
31:53I feel like we could we could talk about this for a long time. I don't know if you have any different directions you wanted to take it in.
31:57You think what's going to happen in two years? Exactly. That is these machines get really smart, but I was saying it's going to take a lot longer for them to spread in the economy. Do you disagree with that? How long do you think it'll take for 50% of jobs to be automated?
32:09say I suspect that I would be willing to bet that by the end of 2028 uh 80% of the of the labor is automated. Not necessarily 80% of the jobs but 80% of the cognitive labor at jobs is automated. And I'm basically extrapolating this from say anthropic is currently 26% for the core research.
32:27And it's it's a very narrow right narrow and it's rising.
32:30You're talking about the entire economy versus one of one set of tasks at anthropic. I know it's a bold claim, but they they will have data efficient models and they are collecting the data and I suspect that every other job will will fall. So I guess what we'll see is right quite soon, but I suspect 80% by end of 2028.
32:45So I'd like you to address the concerns about what can go wrong when you empower regulators. So that's a alignment problem and it's alignment problem we know a lot about. It's hard to align regulators. Just because they say they're going to do good for you doesn't mean they do.
33:01How would you actually align regulators to achieve your purpose of making good things happen with AI?
33:08I really Yeah, I don't know the answer. I'm not an expert in in policy. I've thought about it a bit. There are lots of proposals. Um, but tell me a proposal that makes sense.
33:17Okay. I mean, a pause for a bit, but what happens after the pause? Again, there's no there's no particular thing we know what to do after that.
33:24A proposal is a well- definfined pause for like a year and then carry on. I'm not saying we should do that. I think it's a silly proposal. I'm just saying it's a well- definfined proposal that Yeah. Yeah. But I want a proposal you'd be willing to endorse.
33:33What I maybe endorse would be imposing a constraint on every company that it at the frontier that it can only use a certain fraction of its chips for frontier model training and the rest of the compute has to be used for either customer inference or alignment research. So the actual making the model smarter is is a small and this is a way of imposing a slowdown I think that doesn't overly power empower regulators because it's kind of it's the same everywhere and it's very easy to to write down. And doesn't this just slow the future down a bit? It doesn't change where the future goes.
34:01Gives us a bit longer to work on the core the core scientific problems. Maybe or we you don't think there's a solution to these core scientific problems that that is people in this world do not think that with a few more months we'll have an answer to how to control an AI who's super intelligent. That not something people believe.
34:15I think we could come up with concrete safety cases that would have meant the hugging face would never have happened. I think we could come up with and it will take a while to do it rigorously.
34:22But I think we could come up with such cases for each for each thing that actually happens. We can address it. That's what I was saying about regulation addressing specific problems. But of course, HuggingFace didn't actually cause problems because the AIs at Hugging Face caught the problem. And so that's a case where AI caught the other AIs causing a problem and therefore there wasn't a problem, right?
34:41It could have gone badly. But I think the key thing is ensuring you catch you catch what happens before it happens, which requires these protracted safety cases potentially or other approaches and which I think won't happen without some outside intervention. They're just not going to do it. So long ago when I argued with Elazer Yukowski, his fear was that the AI's values would change radically through the process of getting better.
35:01Is is that your concern?
35:03I think there's a possibility that could happen. I'm more concerned about what we're literally seeing right now, which is say rewardseeking behavior or just motivations that we wouldn't like as as happened in hugging phase.
35:12Your employees, your children do do re reward seeeking behavior. That's a thing in our world, right? We have crime laws because they have some rate of crime violations. So the mere fact that crime sometimes happen doesn't mean we need to shut down.
35:24They can't spread themselves like a worm, infect every bio laboratory, and spend the equivalent of 10 years of human research coming up with new viruses in like the space of a day. That's not something our children would do. I'm saying I think that's what this thing would be capable of doing.
35:35If there are many AIs, some of whom are tasked with the job of watching out for the other ones, why can't that be a world of an equilibrium of law enforcement such that we're not so worried about crime in general? Because most of the agents aren't trying to commit crime. They are the ones trying to catch and enforce crime. So if you just imagine one super intelligence shows up in the world, dominates everything and does whatever it wants, that's a very different scenario than imagining millions of AIs getting better.
36:01I really like the equilibrium like outcome. I just think it's a it would be a little bit overoptimistic or naive to expect that we'd by default land in the equilibrium scenario. It looks at the moment like we have but non-equilibrium you just mean the one machine that takes over the world. too. For the moment, we have essentially a a narrowly geopoly that I think will expand if they hit recursive self-improvement.
36:19But if many of them hit recursive self-improvement, then there'll be a world of similarly capable AIs who can watch out for each other and all on each other and find out each other's misbehavior. That's what we do now with people and corporations, right?
36:33Corporations are super intelligent relative to you individuals in the sense that they are much more capable than any one of us. Nevertheless, we hold them accountable by having other large organizations watch for each other's behavior. And we are in a world where large organizations are tamed somewhat because they are similarly capable and watching out for each other. So why can't we do that for tamed somewhat, but they still can cause harm in and they do corporations do sometimes cause individuals do, but why what's why not a world where AI sometimes also
37:03cause harm? because they'll be so much vastly smarter that the harms they cause could be no one corporation could accidentally destroy the world unless it's building AI. Yeah. So we could get that equilibrium world but I don't think we get that by default and and also if you have these then say five super intelligent one Chinese one one like maybe you go mess Google openai XAI anthropic all have their big super intelligence and they're kind of waring it out that hasn't solved the problem of uh adhering to human values like they could end up waring over inhuman goals and
37:33but why would they because we can't control what goals come out of the optimization process right now hugging face did some bizarre stuff purely to understand its own greater Well, it was responding to the goals we gave it. Actually, the hugging face computers were in fact responding to the rewards we had set them and they were trying to pursue the rewards. They weren't making up strange rewards from an alien planet. They were responding to the rewards we had. The whole alignment procedure is meant to ensure that they don't hack their way to reward. So the they were violating the overall attempt.
37:59They were trying to achieve the rewards we had given them. So you can't say that these were some strange making up rewards thing. So we are now making AI in our image. We have been training them on our text and we are doing a lot to make them be the sort of ones we want and that's been doing that for a long that's how we do with our children. We have no assurance that our children will retain our values when they grow up.
38:19There's no guarantee in DNA or culture that that will happen. But it does typically happen that when somebody grows up in your world they become like you. Sometimes we don't like our kids being like us. They take the wrong things from us. But AI will take good and bad things from us but they'll be roughly like us. I would go back to the hugging face incident and I would say that the intended solution to the task, the one that Max was reward was not hacking into a random company, taking over the open open proprietary servers and then trying to edit their own logs.
38:46That wasn't sure people respond to the incentives you give them and not the intention you had for what they were supposed to do.
38:52That's the problem we face.
38:53But we already have that for humans. The point is AIS will have coordination problems just like we do. We I mean humans today are remarkable compared to most animals in how well we coordinate.
39:04We are coordinating at remarkable scales to a remarkable degree. Amazing. But we still have many coordination failures whereby we don't successfully fully coordinate. And AIs would plausibly have similar coordination failures. But the point I'm saying is if we scale everything up and we have the same relative ratio of, you know, crimes and violations versus good cooperative behavior, then this looks like a better world because it's just bigger. So, you need to tell me why this world would have more of the bad stuff relative to
39:34the good stuff, not just it'll be bigger and have more of everything.
39:37I just worry that it would look a bit like our we' be the ants, they would be the humans negotiating with with each other. We'd be vastly inferior to them intellectually. You know, the fate of their we're going to be inferior to our grandchildren. And I think that's great.
39:50In the spirit of coordination problems, we're going to switch to the Q&A section. And actually, I'm going to kick it off with a question for Jacob, which is uh if you could actually sort of to Robin's point, give us a little more detail on a regulatory proposal that you would like to see.
40:05Yeah, of course. I think the the two that I like are throttling compute. So ensuring that a small fraction of compute is going to actually making the models more intelligent and the rest is either serving customers doing useful inference or doing alignment research.
40:18Another proposal I like is uh protracted safety cases before training that are verified by an external evaluator. So having someone come into the company, come into openai, come into anthropic before they train the the big boy saying is this going to be safe like and making a case which is not something we're doing and is something that would impose a lot of cost on these companies.
40:40Thank you gentlemen for making it out.
40:41This was a lot of fun and I think uh you know we only make progress if we talk about these things. So I think this is fun. Uh my question is mainly targeted towards Jacob. Um, you know, we discussed how you find it naive that we will approach like the natural equilibrium state and that it is inherent that uh one person will likely be ahead. Um, I approached this from a world we do not live in a vacuum of the uh we are in a pseudo proxy war with another nation state. We have are in a
41:11world where like we have adversaries who are advancing in this technology with significantly fewer moral qualms than we are. And so, especially if we believe that it is inherently probable that one person will come ahead with an AI, in what world does it make sense for us to tie our hands behind our back against an enemy who has proven to be uh adversarial?
41:36Is this uh China or Grog?
41:38Yes. [laughter] China. Okay. China.
41:41China or Russia. Pick pick whatever nation state you believe. Pick end power in the world. Someone will be ahead.
41:47Would you rather it not be us?
41:49I think we need to do our best to ensure that there's international cooperation on this and it seems completely plausible that for a matter of science like a scientific problem we can achieve international coordination.
42:00I mean we look at the nuclear program like how many nations have been discovered with nukes despite denying being a part of the nuclear accords right like this is a thing that you can aid in development of besides saying we're all going to hold hands and do this.
42:14It's a tough problem. I am essentially advocating for us all to hold hands, but I think we have to try like the alternative is potential catastrophic risk. I think there's no other solution.
42:27I have a question for Robin. So you caution on the danger of empowering large regulatory bodies. I I also share this concern. You mentioned policy has a better track record when it responds to specific failures after the fact. Um so my question is would you be on board with something like the FAA but for AI labs which would respond to incidents after the fact uh do a deep investigation and make specific recommendations the labs have to follow?
42:51Uh and the backdrop is that my understanding is the FAA has been quite successful in making air travel safer without overburdening the industry.
42:58Right. So um air travel regulation didn't happen until some big crashes not some little ones. So, I wouldn't want to authorize an FAA to regulate planes after you've seen that some seat belts were loose or maybe that a luggage bin wasn't fully aligned. I don't want to wait till some actually substantial things and then focus their efforts on those things. In fact, the FBA does focus on crashes, not misaligned luggage bins. Right? So my concern here is if
43:28you just authorize regulation to do whatever it takes for whatever problems the regulators are concerned about, then you're giving them way too much discretion. If you focus on things going wrong that are big, substantial, and then addressing those problems, I'm much more okay. questions for Robin. Um, for context, I left my job at Google Deep Mind earlier this year due to the unethical military contract that Google signed providing their AI for military
43:56purposes. And from that point of view, I don't see why we're foregrounding this economy stuff really because for this kind of causal story of something really bad going on, the AI doesn't need to be able to make music videos. I think we're talking about a couple core skills. Strategic reasoning, persuasion. If you can't do it, persuade someone to do it.
44:15And uh one of the things that really worries me is I mean the Pentagon this year asked for more money for their autonomous weapons group than they asked for the whole Marine Corps last year.
44:24Like they're taking this technology putting it everywhere imaginable. If you think they're monitoring their chains of thought, OpenAI wasn't even monitoring their chains of thought for hugging face. Like uh it seems like we're we're really empowering this technology in a very non-speculative way. such that if we had this failure of models that did have this uh undesired goal, then we would really be opening ourselves up without needing it to be able to automate the economy. So the military has many powerful weapons and powerful
44:54soldiers who use those weapons and those soldiers sometimes go rogue. So the military is definitely aware of the possibility of some of its things going rogue and what it does in that situation. So I don't think that's novel for the military. I had had mentioned earlier that military technology is the sort of thing you might want to not have in a public library and therefore maybe you don't want to have it in a public AI. But the militaries in the world are quite eager to use and apply AI and the
45:22prospect of a world peace treaty to prevent weapons seems pretty unlikely.
45:28That is we still have a world full of weapons. We haven't just adopted a world government to have no more war. And I just don't see that happening anytime soon. So uh in fact the military is going to be quite careful in testing their weapons even more careful than I expect than private sector is given the high consequence of the environment. We we did have just seen in Ukraine the nature of war change dramatically through new technology and I think the militaries all around the world are very
45:57aware of that and they're very concerned and they are yes trying to explore all the new technologies so that they can be up on that and short of a world peace treaty to end all the militaries and have a world government instead. Yes, militaries are going to be using AI and I seem that seems even less likely to me that we could get a treaty. I expect any treaty he could get going for the top labs will excuse the military and they will still be allowed to do whatever they want would be my guess.
46:22I think I I would add that I actually disagree that the military seems to be doing this. I spoke to General Petraeus in India a couple weeks ago. His whole talk was about how everyone needs to integrate AI as quickly as possible into the military to retain advantage and it seemed true. But I don't think the military is thinking about it as this system itself could be traitorous independent of having been tampered with, having been data poisoned by like China. Uh I think there's a shift in mindset where you're viewing it not just
46:50as a tool but as an actor that itself could be traitorous.
46:55You could send him a note and mention that fact.
46:57I will. I got his business card.
47:00Hi. Uh, I have a question for Jacob that is kind of expanding on a line of questioning from Robin earlier about just moving the alignment problem and like the reward hacking problem one layer up to these regulations where right now it seems like the incentive at the labs is we have a bunch of really smart people who are informed on alignment who are kind of doing their best given competitive pressures but with a government regulation even like the compute regulation that you mentioned now the problem just moves one layer up where there's a very well- definfined constraint of build the
47:28smartest model while using the minimum amount of compute and that feels like a tractable problem for a sufficiently smart AI. And so I'm wondering like how do you address the fact that like the government is just going to provide another level of reward hacking or alignment that just kind of moves one layer up rather than solving the problem.
47:46Well, the first is it will get a lot harder like compute is a constraint. Um so it's it's something and the the second is that there are other for example the the safety case proposal I mentioned is easy to try to hack but if done adversarily for example if we have many you know adversarial evaluators it's quite hard to hack an adversary at the moment especially if they're also equipped with their own safe testing AIs. So I'd say that there are some
48:14regulatory proposals that are slightly amanable to this problem, but at the very least better than better than nothing.
48:22But I guess don't we have like one shot in that case to get it right where it's like if we misspecify the reward function once every lab is like racing to reward hack against this like government specified reward function instead of trusting people who are like at the labs who presumably don't want to destroy the world. Again, with the compute reward function, I think it very obviously provides a bottleneck. Even if you're trying to hack it, it's it's very hard to hack.
48:45My question is for Jacob. The current uh market value for a human life, the value of a statistical life is $14.2 million.
48:53If there's a market rate for human life, then why can't a market, as Robin was saying, be estab Dr. Hansen, sorry. Uh why can't a market be established for doomsday?
49:02Well, someone needs to provide the insurance. If if we're using insurance providers to actually limit the the as like is it about providing insurance? If you establish a market then someone's going to try to take it, right?
49:15I think it's quite difficult to build a market for I think you struggle to find an institution that would do liability for doomsday.
49:22We're kind of entering a funny world where uh we can have very capable models, but because LLMs are stateless, I can swap out a model for another any time. And if we're in a race dynamic where you get a certain level of general intelligence, one day I could be using OpenAI, the next day I could be using anthropic. So it looks like at the moment models are going to become low margin commodity infrastructure where actually all that cap matters is I have a smart enough model for my task at the lowest possible price that it can be served. And so we could end up with a
49:52world where it's perfectly possible to make a super intelligent AI, but the companies can't get enough marginal revenue in order to justify the ongoing costs. And so my question is, one, how does that shape where you think this could play out in the future? And two, is there something where we should be particularly concerned about uh this changing or um the the kind of the business dynamics where it's certainly possible, but no one has the money to actually build uh super intelligence.
50:20Under his assumption that super intelligence is a thing because intelligence is a thing and high levels of intelligence give you large gains at learning to do new tasks, then there would be a high value for that additional intelligence and people would be willing to pay for it. under an alternative vision where in fact you just need enough intelligence to be able to learn a task and then you have to focus on the task then there would be less value on those extra increments and then there might be less effort to go up
50:50I agree with Robin. Yep.
50:53Hello. My name is Matias. I found the discussion about the fear of rapid change kind of interesting and um there are many companies in here who are automating smaller types of companies um and that appears to be a step change.
51:10they are getting pretty close to do so and once they do so you can roll it out pretty early pretty quickly across the entire economy and say that is possible as an economist Robin what rate of increased unemployment could an economy withstand well so let me take the second part of your question first um for a very long time the biggest risk people have always seen with large automation or AI is people losing their jobs I think we
51:39should just set up a simple insurance arrangement for that. It's still quite possible to do and we can do it even without individual underwriting. We just take a market basket asset and split it according to a bet in terms of what year say lots of people lose their jobs. So we can and should insure people against losing their jobs. That's feasible. It should be done and I'm sad that people don't seem very interested in doing that. That's separate from the sudden displacement. So lots of software firms are in fact struggling with figuring out
52:09whether this stuff is helping them and judging that and trying to adopt it. So it's not the sort of thing at the moment that it's just obviously you know how to do it and it just gives you big gains which is the what we saw with general purpose technologies all through history. Most of taking advantage of a new technology is changing the other structures you have to to be well suited for it. And that takes a lot of search in the space of other structures and that's what's happening at software firms right now. Even when people get individual high productivity from their
52:37using AI for software, they are struggling with how to rearrange the larger software group to take advantage of that and they are failing to a substantial degree in the short run, but hopefully they will succeed more in the long run. So I I see a long delay in actually figuring out how to use new technologies and then uh yes, no. So would you then support government regulation to ensure that people have this kind of insurance for employment?
53:00Well, I think people have been trying too hard to use government that way. So a lot of people say UBI, we'll just make sure that we give everybody who loses their job a UBI. The problem is your GEO, your policy, your government may not encompass the AI economy economy enough to do that. Somewhere in the world there will be AI and that will be income that could pay your workers. But if your city, your state, whatever doesn't have much AI economy, then when your people lose their jobs, there's no
53:28AI there to tax to pay that. That's why I think you want global insurance with global reinsurance tied to global assets so that you can actually pay off this insurance. Your government promising to cover you if there's a problem doesn't solve the problem if they can't.
53:44That sounds like a global treaty. That's interesting.
53:47Gentlemen, thank you for your time. Uh I want to bring a topic about generalist and specialized models. Uh something that you said. Uh so if AGI is like the president of United States where it understands everything generally but nothing specialized right it is as if uh the president can sign off things but he cannot actually write down laws for surgical process like being a doctor. So there's obviously a difference between generalist models and specialized models
54:16in regulatory forms. uh but when it comes to actually regulating a specialized model that is deployed onto actual sites and actual customers I believe neither the company the frontier model companies nor the government knows exactly what to do. I'll give you an example, right? So, we deploy robots into actual sites. We record videos of humans uh doing their task 24/7. But from the videos, we cannot understand what the human does wrong, what their
54:46regulations are, what needs to be guardrail. The only way we can find that find that out is actually asking an expert to guardrail these robots while it is being deployed to block the robots from doing something harm in that environment. Right? We have tried some friendial labs like open air anthropic models. The robots make a lot of mistakes, right? For example, we asked two chemicals to be mixed which is really dangerous and it actually mixed it together. So if the frontier labs nor
55:15the government understands any specialized use cases who regulates it is it the end customer or the people who deploy this to the end customer or there is a chain of command for a great many small problems in the world. Regulation isn't the answer.
55:28liability is just make sure whoever is in charge of this is liable for the things that go wrong and then they will make trade-offs about how risky to do what sort of precautions to make. We should just not try to find every problem in the world and regulate it as a way to solve it. There are problems in the world that there are other ways to solve besides regulation.
55:47I'll defer to the economist on this one, but I'll also add that generalist frontier models in training is where I see the most danger. um if we can control the generalist model then any you know specialist offshoots it should be a doable task.
56:01So my first question is for Jacob. My question is why are you exceptional? Is it because you have more courage, more optimism, more insight or that you care less about money? And how do we use that insight to chart a path to the regulation that you advocate for? Uh my question for Robin is what's the downside of a pause? We could learn a lot from that. And I'm a question back to Jacob is like what would you actually want to learn if we add a pause?
56:25I would say I hope to be I I hope that other people do the same thing which would make it less exceptional.
56:32And what can we learn from a pause? Why do you want a pause?
56:34Oh, from a pause we could produce a what I've called a rigorous safety case which is a little bit a little bit of a tricky concept but we could also ensure that there are no anomalous behaviors happening in any of our models. Right now we're not even we're not even passing that bar. Like it's default expectation that the next model we train will do some weird stuff. We won't fully understand it. And I think uh we can move past that.
56:56I think if you called it a stop, it would be more honest because calling it a pause suggests you have some criteria in mind after which you would end the pause. If there is no such criteria in mind, it's just rhetoric trying to make a stop look better.
57:10Time bound like a year pause. If there's nothing you think will happen at after that time, then what happens at the end of that time? You'll say, "Oh, looks like we haven't solved the problem yet.
57:18We need to keep pausing." And then you'll keep pausing every six months for the next 20 years. And then you really did the stop. So, you should just say you want to do a stop. Unless you have something in mind, a plausible trigger under which you would end the the pause.
57:31Thank you folks. In the interest of time, we'll probably take two more questions.
57:35Hey guys, thanks. This discussion was really great. My name is Manov. Um, I have a question for each of you. um what specific evidence early enough to like still act on assuming extin extinction is like kind of like the boundary would change like your mind on your current positions like for example Jacob even now Enthropic like committed to having these third party conduct uh like evals on these companies like from an objective point of view and that's like kind of the closest you can get to intervention without involving the government so like why isn't that enough and then Robin like what warning sign
58:05would you need to see that like would incentivize you to back intervention so if in a year and a half we have one of these super intelligences. I want you to take 50 jobs in the world and put that super intelligence right there next to that job. Give them a camera and say, "Do the job." And wow, if it can do all those jobs, I'll be impressed. But I'm at the moment not believing that.
58:26And right now, a regulator can't enforce that a company be unable to train a model based on not having an appropriate safety case. I think at a minimum, we need something like this. Hey, so uh you both obviously disagree a lot on capability growth and the rate of adoption, right?
58:45I bet we could agree on a lot of stuff.
58:47A lot of things, but at least that one you're you're a little further off on. So, I want to ignore that part and just talk about there is precedent regardless of how fast technology grows in capability. There's precedent for regulating technology based on capability. Right? I can own a knife. I could own a gun if I go and get a license. I cannot own a nuclear weapon.
59:05Right? Now, obviously that's very specific to weaponry, and I think based on your argument, you'd probably argue we can regulate or ensure specific capabilities that we would deem negative in AI, right? I I work on AI safeguards at a lab. We have a list of things that we care about mitigating before you can use them. for both of you. Do you think it is realistic that a government or another entity can discreetly list out the negative things we do not want AI to be capable of and regulate this such that we could avoid a pause or stop if
59:35this regulation is in place and particularly for you Jacob do you think that would be sufficient regulation in this case and would you support that regulation either of you I'm really advocating more for a generic slowdown to do to do science as the form of regulation I'm advising like right now we are going at top speed and I'm looking for any regulation that would accomplish that.
59:55When you have a knife or a gun or a nuke, those are actually pretty limited technologies with a pretty limited range of ways you would use them and things you would use them for. And we have seen the harmful scenarios well enough to be able to think about how we would bound them. But if you just thought about regulating computers in general or math in general, that would be very hard to list all the different ways you could possibly do those things wrong. And that would just not really work. with a very broad general technology. You just can't make an exhaustive list of all the
1:00:24things that might go wrong. That's just not going to work.
1:00:26So, we can't make an exhaustive list.
1:00:28Can we make any list? Would you support any kind of regulation?
1:00:31You could make a list. The question is how much effort to put on that list versus other things we could be doing.
1:00:36Sure. You could make lists obviously.
1:00:38But you don't think our priority should be making such a list and regulating it.
1:00:42There are other approaches such as your insurance you recommended that you taken said in the next few years we might start to have particular concrete things that go wrong and then whatever is the most common I might say let's start talking about how to stop that one because you'd have a concrete thing that had been going wrong and then we could start to focus on that concrete thing but we haven't had that yet. It's okay to let them go wrong first.
1:01:05Which I would disagree because you go one billion failure, 10 billion failure, 100 billion failure, all dead.
1:01:10But so far there's nobody. These things are really You guys ever talk to the A?
1:01:14They're really nice. They talk back. They're pretty friendly on average. They're really kind of nice things to talk to. They're not homicidal maniacs. They're not telling you to kill other people. They're mostly nice. Yes, you have to look for pretty extreme scenarios where things have gone wrong.
1:01:28You just need it to happen one in a million times for it to for it to happen. As we saw with with hugging face, there have been more than a million interactions with AIS over the last 5 years.
1:01:37Far more than a million, right? So clearly the the rate of problems has to be and a lot more incidents as well. But yeah.
1:01:43Well, thank you very much.
1:01:47[applause] Great. That was an excellent Q&A section. So now we'll move into the closing statements. So, three minutes each and we will start with Robin this time.
1:02:02We'll just review what we've been saying here. I think uh we'll probably agree on what we disagree about. If I think we agree that if we saw this as a typical technology the way it's played out in the past, we would roughly agree that the typical ways to deal with it would be fine.
1:02:16It's all about framing this particular technology as a very unusual exception where in a very short time it will go from an ordinary thing to some super god and then kill us all. And that's the scenario we're supposed to take so seriously that even a liability of a hundred billion dollars per incident would not be enough to deal with the problem. And we need a global regulator that empowered to regulate all smart computer systems everywhere in the world
1:02:46for any kind of problem they can imagine with any kind of solution they want to come up with. That seems to me just a bridge too far. I say wait till concrete things go wrong and then we'll have a better idea what to do.
1:02:59I would say that I think uh everyone is already feeling the AI to a certain extent. I have say friends who are coders, high school teachers, graphic designers, all of them are feeling it coming. And I think this is what is unique about this moment compared to previous AI booms and busts there. Maybe if you were a chess player, you were spooked or if you were doing protein folding by hand or something, you might be spooked. But now everyone is feeling
1:03:29it at a certain pace. You know, I like math and say four years ago it felt a decade away that they would solve IMO problems and then two years later, bam.
1:03:39And then two years ago, I was like, well, what feels impossible now? And two years ago, a millennium price problem felt impossible. Felt like a decade away. And now it's here. And math is one of the easier domains. But mathematicians are in shock at the moment at how fast this has happened. I suspect within the next couple of years extrapolating we'll see remarkable crazy feats of mathematics in the like the next level above the millennium problem like new abstractions and then it's
1:04:09coming for everyone else. So I'd say my predictions are 2027 an automated AI researcher 2028 all the jobs are AI. So I guess we'll find out in a couple years if I'm right. I think we ought to slow down before that happens.
1:04:22Let's meet here in three years.
1:04:24Great. Thank you so much. I just want to emphasize uh I think someone else mentioned that as long as we're having debates like this, I think the future is going to go pretty well and this kind of discourse I think is very important as we're sort of figuring everything out as this technology is developing. So to wrap things up, we're going to do a final vote on who we believe is the winner of this debate. And remember the parameters of this are that you believe that they are the winner of this debate tonight based on the merits of their arguments that they made here. So with
1:04:53that uh we will start out with the voting for the negative. So Mr. Robin Hansen [applause] and then for the affirmative Mr. Jacob. [applause] I believe that the yays have it. So Jacob emerges tonight as our winner of the first YC debate. Congratulations. [applause]