0:00Today, one in 25 American teenager says that they have had a deep fake nude produced of them. I'm not saying one in 25 tell you I know somebody to whom this has happened. One in 25 tell you yes this has happened to me. If you want fast collective innovation, if you want people to have access to the tools they deserve, if you want people to trust the things that they run their lives on, give them good shared open source foundations. policy without tools is
0:28basically poetry. It looks great. You can recite it, but you can't really protect people with it.
0:34In June 2018, Twitter bought a trust and safety startup called Smite and abruptly cut off its other customers. Platforms like GoFundMe and Task Rabbit lost their anti-abuse tools overnight. It exposed a bigger problem. Safety infrastructure was concentrated inside a handful of companies, leaving smaller builders dependent on tools that they didn't control.
0:57Dr. Kami Francois has spent her career investigating online harm and building ways to combat it. In February 2025, she launched Roost to make safety tools free, open source, and available to anyone building an online space.
1:12As a kid in France, Kami wanted to be a space baker, baking croissants on Mars. She's chocked that up to being French. Today, her ambition is to make building safely as accessible as building itself.
1:25Her colleagues describe an anti- athoritarian streak drawn to frontiers over ffts. That may help explain root's approach, sharing technologies that others have kept proprietary and giving communities the tools to protect themselves. Kami Francois is the founding president of Roost and a professor at Columbia University's School of International and Public Affairs.
1:51Welcome, Kami. Your colleagues describe you as having an anti- athoritarian streak, more pirate than institution builder. Does that sound like you? And what were you like growing up? I'll talk to my team about that in an authoritarian way. I hope.
2:13No. Yes. Yes. Of course. I I I recognize that description. Um I I do think that I am able to work with and within institutions.
2:27Um but yes, I like the energy of uh doing something together in a collective. um maybe with a little bit of a pirate spirit. Um and when we were preparing for our very first um roost team retreat, I actually found a book which is great called Be More Pirate that looks back at the history of the glorious days of piracy and think through okay there are institutional
2:54lessons to take from those days. Um, and there's a way to rethink about uh that uh moment in time as a moment in time that brought a lot of advances in equality and democracy. And so uh maybe uh maybe there's a different lens we can apply to uh to our understanding of piracy. But jokes aside, I think that um I've always uh been part of the open source and free software movement. Um when I was a student, I founded uh
3:23students for free software friends and then sort of served on on the board of the global organization. It's actually how I met my my husband. And that organization no longer exists, but back in the day we were referred to it as Baby Pirate Hacker Club. So yes, I I can see I can see where that comes from.
3:40Yeah, exactly. I mean, anti- athoritarianism, I'm I'm with you.
3:44That's great. Uh, and it seems like everyone is talking about online safety, privacy. You were way ahead of the curve on this. You are building Roost, you mentioned, which is a nonprofit that makes online safety tools free and open source. For someone who is hearing about this for the first time, uh, who are these tools for and and what do they help people do?
4:05Yeah, they're for everybody. And Bree is a nonprofit. It's also a movement. Um, and so we do this with a lot of people. We do this with large uh AI labs but we do this with small builders. We do this with philanthropic organizations. We do this with many different types of institution. And essentially if you think about safety we have at least three problems.
4:25The first one is I think it's fair to say that tech and AI has progressed faster than our ability to keep people who interface with these technologies secure. notably when it comes to kids and children.
4:39And so we have an innovation gap, right?
4:42We need to fast forward our innovation and progress on safety tech. The second problem that we have is a sort of a distribution gap, right? Um there are a lot of safety tools that are okay but people don't have access to them and there are many more builders who are building new applications, new websites and who want access to the tools to do things right to protect their users, to protect their communities, to do safety by design, right? That that's a that's a
5:11a real problem that we have. It's nonetheless a good news, right? That means there's more appetite for safety.
5:16And then of course the third problem is there's a little bit of a crisis of trust when it comes to safety technology. It's because it has long been very opaque, right? So people tell you like, hey, we're using technology to keep you safe and you're like, okay, what is it? Can I see it? And it's like no. And so of course we can change that model here. Open source really is a method, right? Where we can accelerate innovation, make sure everybody who wants those technologies have access to it. And then of course where we can say, hey, here it is. here's a safeguard. Go
5:46audit it, modify it. If you don't think it works for your use case, go make it work in your language, your context, and then if you make a better version of it, just go distribute it or sell it, right?
5:55Like for that matter.
5:57You've spent a number of different years both as a pirate entrepreneur and um professor and then also, you know, years investigating online harm, work on informing the Senate on on disinformation.
6:14Um, was there a moment that convinced you to build roost? Was there a was there a kind of a ah this would be a way of changing this ecosystem? What was that moment and uh how did your experiences lead you there?
6:28I don't think there was a moment. I think that has been slowly building up and frankly not just for me but for the field and if I think about my own trajectory in safety of course I have taken from all those different experiences ingredients that you find in Bruce right so when I was studying information operation one of the thing that you very quickly find is information operations are crossplatform right and so you find them spreading on
6:57smaller bulletin board weird little forums apps that nobody's ever heard of.
7:02And when you sort of unravel this, you have a lot of people coming in saying, "Oh, we did not want to use the school board in this way, help us, right?" And so you sort of understand that any sufficiently smart and motivated adversary is going to take advantage of the fact that there are many surfaces and smaller surfaces and not everything is protected. Of course, that matters a lot for the theory of change at Roost.
7:24When I was at Niantic, that was a really formative experience because I had a great CEO, John, who was very supportive of our safety efforts. And of course, Niantic had Pokémon Go. A lot of kids play Pokémon Go. And when we said, okay, we're going to build a really good robust infrastructure for safety. I went and I sort of knocked on doors and looked around like, what's the state-of-the-art? And I did not like what I found. And the state-of-the-art was like, "Yeah, what what we do is we kind of everybody builds their own
7:52halfbaked stuff and then you you buy a few good vendors and then you duct tape everything together and that's what everybody does everywhere." So I was like, "Okay, well that sounds nuts.
8:01There has to be a better way." I was coming from more of a cyber security background where of course there's a different approach, right? like there's a shared infrastructure of open source tools across the field so that you can focus your resources on building the piece that makes sense for you but you don't have to sort of fully reinvent everything from scratch all the time.
8:21Um, and so I started thinking about, okay, well, let's try to figure out if those tools that everybody's sort of cobbling together, are they the same for everybody? Like, is it are we doing this because people have such unique needs?
8:35And so I started interviewing a lot of safety engineers across the field and they were like absolutely they're the same. Most of the infrastructure is steady. So we did this project to sort of document what are the core components of safety infrastructure. We published a piece called a dire framework that goes for detection, investigation, review, and enforcement. But really what engineers were telling us is, yeah, we get staffed on something, we built 30% of a workable thing, and then we get assigned to another project. And so everybody's sort of constantly
9:05rebuilding basic foundations, which is really frustrating. And so we said, okay, great. There's an opportunity to change the way we do safety in tech, to sort of rewire the way we do safety in tech. I feel like everyone's talking about there's a lot of other issues and discussions around AI, but a lot of people are seeing AI as sort of social media 2.0. We got social media, we were so excited, there were these harms, some of them to children, some about misinformation, that we feel like we didn't sort of attack soon enough and people are worried that it's going to
9:34happen with AI. Like, do you think that if Roost had been around 10 years earlier, the landscape would be different? And like, where do we go from here? I actually think about social media as an exception to the rule of how we build tech as a blip in the timeline.
9:51That's just a bizarre happenstance, right? Because if you take the long arc of tech, the way we built the internet, the way we built the foundation of tech around us is on open protocol and on open source software, right? TCPIP was a open protocol. um Tim Berners Lee donated the web to humanity, right? And so the reason why we can innovate and then keep tech relevant and keep tech responsive to our needs as a specy is because we do this in the open.
10:21And then there's a sort of small era of social media where we kind of forget about all these lessons and it's bizarre, but it's a fact that we've inherited all of the safety tech that we have from that blip in the timeline, from that era. That's why it's brittle. That's why it's hyper centralized.
10:36That's why it's opaque. And so what we're doing with Ruth, frankly, is sort of bringing back all those normal principles of how we build good tech into the world of safety. We're not really inventing something new rather than sort of like getting back to the roots of the internet.
10:52Exactly. Right. Changing the approach to something we know works, which is okay. If you want fast collective innovation, if you want people to have access to the tools they deserve, if you want people to trust the things that they run their lives on, give them good shared open source foundations.
11:08And do you think the leaders of the Frontier Labs are learning from any of those mistakes or like what what mistakes are they still repeating?
11:15I think some of those leaders are building this new open source foundations with us. So last year for instance, we did a big effort with OpenAI. Our team worked uh closely with their scientific and engineering team to take one of the core piece of their safety stack um a model that was doing a lot of moderation uh behind Chad GPT and then Sora and to actually share it with everybody to put it out in open weights.
11:45Now that's significant for a number of reasons but the first one is prior to that it's not like you didn't see big companies share safety innovation. But a lot of that sharing look like oh we're going to make this available to everybody. Usually that that lasts for the duration of a really good marketing campaign and then of course like you know priorities change and then people move on and so things are not maintained uh people are struggling to get access things sort of fall behind and then of course at any
12:14moment they can pull the rug right they can say actually we changed our mind but open source isn't that open source is a one-way door right so when openi says we are actually going to share this model under an apache to license and we're going to put it on hugging phase that means tomorrow they changed their mind they can't get it back and so people know they can build on it right so we did that and they made that statement at the time saying that they believe everybody should be free to
12:41study modify and use uh critical safety tech which is kind of fun to see open eye quoting uh Richard Stallman but uh I love that and after that we also did a really nice hackathon uh working with both open eye and hugging face so we invited people saying okay look this model has been shared come test And so we saw people who were really exciting to immediately use the model. For instance, today this model is used by Wikipdia, right? We will also uh you know keep in mind that large important uh tech nonprofits also deserve safety
13:11tools. But we also had a bunch of people come and break the model which was really fun to see. I remember there's a team who I think was building um recipes for meth and ecstasy in Latin. Uh so yeah that's sort of like what uh we you know we see some of those large lab being like okay yes we think we want uh you know we want to tackle safety a bit differently. Now I want to go into a specific area as an example in a second but let's first come
13:40back a little bit obviously I'm not giving you the recipe of meth and lab don't don't insist I want it in French um the um uh one of the things like when you think about like I think there is a lot of fundamental important about open source as you know I was on the board of misilla for like a decade and and you know LinkedIn actually open sourced a whole number of projects and I think it's a generally good thing to do I think it's also so good on on certain patterns of safety and security. Um, and I think the foundation of the internet
14:09on it. I don't think it's actually only social media that creates the more turbulent stuff. I mean, we have the mobile internet with Apple now. We have a bunch of like there's like kind of what's going on behind YouTube, you know, within Google and so forth. So I want you to kind of elaborate a little bit of what is this weave look like and how do we bring because it's not a disagreement that open source isn't an important thing here to elaborate this point of view but it's like given a
14:38complex environment what is the role that you see open source doing and then I'll get to a specific commercial example question I think here open source has a unique ability to help us, as I said, reshape how we think about safety, really accelerate innovation, make sure that a lot more builders have access to the tools they want and need, um, and
15:05enable a system that can be scrutinized and be adapted to a lot of little communities. When you think back to the social media era, we talked about the sort of perils of hyper concentration. There's another peril which is this idea that I think we held a little bit too long that we were going to have big global public squares with rules that work for everybody.
15:28Well, I don't think this panned out exactly the way we wanted. I think the reality is you're going to have a lot of smaller communities that have rules that make sense for them and that's important. That's pluralism, right?
15:40That's the world that we want to live in. If I think back about the difference between today for instance and say like 10 years ago as you said I'm a professor 10 years ago if I looked at my students online lives they were quite concentrated on a few platforms right so for instance they may have like they use they use back in the days what's called Facebook you know Facebook for uh school work and also communicating with family and all that but today I think what we see is uh people use a lot of different smaller
16:10platforms for a number of things right?
16:11They may actually have a WhatsApp group for something, a signal group for something else. They might use a Discord server to coordinate about something and then you know whatever, right? So this multiplicity I think now of online spaces both in social and in AI also means that we need to be able to give all of these communities the tools that they deserve to shape what safety means for them. Right? It's also bringing back pluralism which is something that maybe we had lost sight of a bit during the social media era because one of the things I know you've
16:41tracked is you know Twitter bought Smite in 2018 immediately cut off all the other customers demonstrated how brittle this is when you have a commercial system. So Roost is a offset to that in in part.
16:55What are the things in terms of has is Roost the only one that's doing this?
17:00Are there more? you know, is Roost trying to encourage things like smite or other things to be open? Yes. Like what's what like h how how does the dynamic play out? That's part of like the specifics within a commercial environment.
17:13Okay. So, it's let's talk specifics of it.
17:15I love the story of Smite.
17:17So, Smite was um a little startup that provided safety technologies to a bunch of companies. Uh one of them was Discord. And Smite, which in fact is a rules engine, got bought by Twitter because Twitter was like, "Yeah, that's very helpful for what we do. I get that." And then Twitter shut it down.
17:36And so a number of companies, including Discord, was like, "Oh, we can't protect our users in community. We just got the rug pulled super superh brutally from under our feet."
17:45And so they did what they had to do, which is they did the very boring work of rebuilding everything from scratch. What they have rebuilt from scratch they have donated to roost.
17:56It is the foundation for one of our core software project called Osprey because discord was like nobody needs to go through that again.
18:05I am going to open source that rules engine. And the second they did that right we worked with their engineering team because actually it takes a minute to to open source an internal system. Uh blue sky came in and said oh that's great I want to use that. So they started using this and as a result they started improving this. And so Discord said, "Okay, well, we want to use the branch that everybody is improving. We want to benefit from the collective innovation of everybody who uses um Osprey." And so Osprey is this amazing
18:33story of like this is what the industry learned from Smite and this is what Discord decided to donate. And now this project Osprey um is actually used by many different platforms and organization at scale to investigate fraud and scams and coordinated harassment. Um and it's it's great to to see it uh take a life of its own. Yeah.
18:57Yeah. It's also it's also fun for for Discord because um well as I said like they benefit from the collective innovation and then they were able to spot really good engineering talent too and so it becomes fun because all that really good engineering safety work also becomes visible. Right. It's a morale boosting for the people who work on this and then also it's helpful for careers.
19:14Yes, absolutely. I mean every engineer wants to work on something and millions and millions of people are using. That's the benefit of being able to be part of that community.
19:22And I think it's also, you know, Discord, it's really important to have all the different actors in the ecosystem. You need the researchers, you need the professors, you need the people who are in the companies. And like Discord was uniquely suited to be able to do something because they had had this rug pull. And we talked about you being a pirate. So you've mostly worked sort of from the outside trying to keep it, you know, under we're only talking about pirates here.
19:46Um, but you also were inside of Niantic, as you mentioned, you know, accountable for outcomes in real time, you know, at an actual company that's pushing deliverables. What like what did that teach you that sort of the researcher being outside of a company couldn't have taught you about this space?
20:02I think it taught me how much tools matter. And if you take a step back and sort of look at the overall conversation that we're having on safety, there's been a lot of discussion on rules, right? What should be acceptable and not acceptable? And I understand that focus because I think rules are really the most visible part of the pyramid, right?
20:20It's the thing that we can all look at and sort of negotiate together. I think I learned at Nandic that policy without tools is basically poetry, right? It looks great. You can recite it but you can't really protect people with it concretely. And so the importance of those systems, the importance of infrastructure, the importance having multiple systems who you know will be interoperable and will work together because a lot of important context that
20:48you need to protect people also get sort of lost in translation when there's a hands-off between one process and another process maybe one internal tool to another internal tool. I think that um I got a unique sort of very uplose and personal perspective uh when I when I built a trust and safety function.
21:06Yeah, actually tooling is really important and it's one of the things that academia usually doesn't get because it's not quite the way that academia operates.
21:13That's an excellent bring into the academic environment generally not just in safety not just in open source not just in right um and you know R was uh incubated at Colombia University right at the Institute for Global Politics. I think we we forget that uh academia can also incubate amazing tech nonprofits, right?
21:34We we look at big uh business schools and all the startups they incubate. But there's actually a nice tradition of amazing tech nonprofits being incubated out of academia. I think of like Scratch uh love Scratch coming out of the MIT, right? So like uh universities here have a role to play and yes, if they focus on boring things like tooling, uh we can go really far. It will help. Um so I saw that your ABC framework was cited in anthropics uh report on AI misuse. So
22:04one first briefly tell us about the framework and then the question is does generative AI break the premise of the ABC framework because now that you can have a single actor you know synthesize actor behavior content all at once is that does it does it work or does it need to be updated? So first framework and then that question.
22:26The ABC framework has had a a fun uh life uh since it was initially published I think in 2019. Um initially uh I was looking at information operations and then sort of more broadly at moderation and ABC here stands for actors, behaviors and content. And it simply says that essentially trying to figure out who are undesirable actors on a platform. For instance, convicted child
22:55sexual offenders, what are manipulated behaviors on a platform and what is undesirable content. It's actually three different exercises and require uh three different approaches that will draw on different disciplines, right? One will be closer to borrowing from spam and the other will be closer from boring to cyber security. So it was just trying to sort of put some order around those three things that we tend to conflate and focus on content where you can see
23:23problems where there is both a problematic actor and a manipulative behavior and undesirable content. But also again it's helpful to sort of separate those three those three things and think about the tools and approaches that are that are appropriate for each.
23:38Um it was very helpful in social media days to uh help uh sort of reorganize uh uh moderation functions where a little bit of the cyber security mindset would also come in. And it was interesting because OpenAI I think were the first ones to also in the sort of AI era to also site the ABC framework. And my talented colleague Ben Nemo who's at OpenAI said actually I think we can do
24:07completions for C. And so he said the ABC framework still works in the context of generative AI. It's just like actor behaviors and completions. Sometimes you know the AI will complete something and you're like that is undesirable. Mh.
24:20Um, and so I think he he's the ones who sort of like broughten in into uh into um the AI context and I was also very pleased to see it cited in the uh anthropic report. Yeah.
24:31Well, so if we're digging in to sort of this safety work, you have detection, investigation, review, enforcement, which of those is hardest to get right?
24:40And we're So one, which of those is hardest to get right? And then we're talking about perhaps it's most important for there to be safety among the big actors that are serving millions, tens of millions, billions of people, but there's a long tail. And so that's also the problem that smaller actors can't, you know, they're underresourced. And so where when the underresourced folks come in, what do they skip first in those four things?
25:03If you think of detection, investigation, review, and enforcement as sort of a pyramid, right? the um or an iceberg rather the exposed part the thing that everybody sees and likes to comment on is detection right so the question is always like oh do we know how to detect this harm it's it's a helpful fun conversation I'll get back to it in a second but in fact once you've detected the harm there are a number of things you have to do to be able to take action and um at root we
25:30focused on sort of building those boring tooling infrastructure pieces is so that people have the means to take actions when something is detected. So we talked about Osprey. Uh Osprey is an investigation tool. It allows you to say, "Oh, I can describe an activity that I don't want on my service by its pattern. Please real time go find the pattern and help me take action." Super
26:00helpful. for instance, if you want to root out fraud or if you're dealing with um sadistic online exploitation and you have a signal that you want to be able to detect at scale. So that's Osprey and as I was sharing this is this is coming from um uh from Discord. The other big tool that we have and that people are using is called coupe.
26:20Coupe tackles review. Now, Coupe came into Ruse through also a bit of a bizarre uh story, which is um we bought a startup, which is a funny thing to do when you're a nonprofit.
26:32Nonprofit and university.
26:34Exactly. Um there were two super talented engineers from Meta who had built a super little startup which was a really good uh console that you can use for review and some enforcement, right?
26:46Because for instance, say you want to detect if child sexual abuse material is uploaded on your services.
26:53If you do detect some, you actually have to be able to review to ensure that you know what this is going on. And so for that, you really want wellness features. And then if you do find that indeed this has happened, you have an obligation to report it if you're in the US to the National Center for Missing and Exploited Children, NECMAC. And that report is actually quite hard to do.
27:12So we bought this startup and we spent a lot of time working to open source the core technology and to really add on to it.
27:21It was quite lucky for us that some of the former customers like notion decided that they were in it with us on the open source journey and really helped sort of like take that seat that base and sort of grow it into what it is. We re-released Coupe a few months ago as the world's first end-to-end tool that allows for the detection, review, and enforcement of child sexual abuse material, fully open source. That's
27:48amazing because it means that now people who've always wanted to do this work can do it without having to knock on 99 million doors and pay lots of budgets.
27:59And what we're hearing is that it has sort of met its audience of again like people who were willing to do that and didn't have the tool. So I can go into into many sort of stories for what this looks like in practice, but the point is we often underestimate the tooling needs that are not just detection on detection. There are still things that we are not very good at detecting.
28:26At Roost, we have the Roost model community. It's a really nice community where people come and actually share their safety model. So, we talked about OpenAI doing this with us. A few weeks ago, Mistral said, "Hey, actually, we also have a model that we use for safety. We think it's quite cool because it's bring your own policy. It's multimodal. It's called shells trial.
28:45We're bringing it to the root model community. That means they have to do office hours and the community comes and say, "Okay, I'm testing it in this way.
28:51I think this is working. I think this could be improved." D everybody shares the feedback and then we sort of continue iterating right there's also areas where nobody has a good solution and that's where we as invest in scientific partnerships uh with independent rigorous sort of scientific and academic organizations where we say okay here we're going to need a big collective effort and push to actually bring some new forms of detection into the world. Can I just ask do you you
29:19know you said like you know thankfully notion came on the journey and there are benefits like you said discord you know they did it but then they get the best of what everyone is doing because everyone's building off them like that's obviously the whole ethos of open source but are we like relying on the kindness of strangers? I feel like people are so worried that we're like companies are never going to police themselves or they're not going to do it if h how do you make it so every company says hey I'm gonna be a part of this ecosystem.
29:44Yeah I want to figure out the answer to that question. You know, I I think that podcast.
29:50Yeah. I I think that uh leading by example is cool.
29:56Uh I do think that when we were early, the few companies who said, "Hey, we're just we're going to do that. Uh we're going to we're going to invest in this. We're going to put our own resources. We're going to share our own systems. We're going to maintain uh the comments." Um I think they they started something that uh continues to grow.
30:14Right now we have more and more contributions to roost. Um I I want that sort of uh uh race to the top. Right. I want people to look at exactly the benefits that everybody gets by sharing this technology, the benefits that they get back in super talented engineering, the benefits that they get back in trust. Right. And of course I want more people everywhere across the industry to say hey actually I also want to share my tools and I can see that there's something that's slightly better than
30:43what I have. So I'm going to adopt that.
30:45The other thing is we're very focused on who else should be part of this conversation. I think Silicon Valley for the longest time has developed safeguards and safety systems a little bit in isolation. And when we think about for instance youth mental health which is a big topic for us there's a lot of really important expertise that hasn't been at the table working in the open allows us to work handinhand with clinicians uh with sometimes victims and survivors right
31:14and to bring a lot more different perspectives than to just do this in isolation in a building in in Mountain View or Menlo Park or uh whatnot right yeah one of the things I think people easily understand or if they pay attention is to why a safety commons and open source is useful in a cost structure because something based off of in a quality structure because multiple people using it and fixing it is good. Say a little bit about how it might also help startups succeed better.
31:44Right. Because part of the thing is like having them grow with it and have it as part of it is actually in fact something that isn't just a oh I'm checking the box to make sure I'm not doing something bad, but it's also helping me succeed.
31:56Yeah, I love this question. is take two different types of startups as a thought exercise. The first one is you're a small startup and you're going to build a social product and you want to focus on making that experience amazing. Maybe you're part of your social product is people can connect and maybe there's a chatbot, right? Um what I see from the younger builders, right, the younger generation that that's building products every day is they have seen firsthand the harms of what it looks like when it can go wrong. They have sometimes
32:25personal experience and they want to build safety by design. They actually don't need to be convinced, right? And they think that uh safety can be an accelerant for them. So they think that you know if they can get this technology then on top of it they can build a better product. There's also markets in which this is going to be the cost of doing business. Right? So for instance uh in you know in certain uh regulations uh you have to uh go and filter out
32:54child sexual abuse material from your platforms or your data set that are going to be used in AI right and being able to access the technologies to do this in a way that you can trust frankly like allows you to compete in the market right so it's a pro competition uh thing generally helps people build better products what about safety startups that's also really interesting Because it's difficult to have a safety startup because as we said most of the companies that you're going to try to sell into organization that you're going to try to
33:23sell into have to pick things up from scratch but if everybody has a shared foundation of tools then it's much easier for your clients to plug into the amazing technology that you are building. So we see already at roost safety vendors engaging with our work.
33:39We have this uh amazing uh company called Zentropy who has been contributing fantastic uh um open weights safety models. You can bring your own policy uh and tune it. And then we have uh another amazing uh startup called Musubi who's actually selling a hosted version of coupe, right? Um, I'm laughing because in my head I think about this this project as Musukoup because Sukup in French means flying saucer. I'm reminded that this is not
34:07official marketing for it. But the point is you have startups who are like, "Okay, this is really cool. We can build amazing services and technology on top of the shared safety commons." Right? So if we do this right, the the shared safety commons is not just a hippie idea. It's also something that's an accelerant to good business, startup, and innovation everywhere. So clearly it's important for safety for us to get it right. So quality is really important. But you also did mention cost that this is actually cheaper. And so
34:36one platform estimates switching to coupe cut its tooling costs by 95%.
34:42Can you dig in there like what made the old approach so expensive?
34:46You know I don't know what they were using. Uh but I think what matters is we are in an environment in which sadly people are you know reducing the budgets that are dedicated to safety. And so we want these resources to be focused on problems that are interesting and unique to those platforms, right? You don't need to use your resources to rebuild over and over basic foundations to build a safety uh infrastructure on. And we
35:13want those resources uh to be focused on the great people who are going to do this work, right? On retaining uh the people who are going to think about the policies, who are going to investigate the incidents. And so it's true that in this context, giving everybody a shared layer to start from doesn't mean that it's totally free in a sense that then you have to host it, right? invest but but it allows people to focus their budget where it will have the most real impact on protecting people uh on a
35:42day-to-day basis which we're we're happy to enable that. So one of the things you know that folks get when they understand open source on the defense is lots of different people can look at it can rattle it improvements are shared across and so there's a set of things that you know traditionally cyber security etc you know the open source protocols for how the fundamentals of the internet they get better but um one of the things
36:10that also is a trade-off is the attackers can then also look at it.
36:14So, say a little bit about the principles about this trade-off between getting stronger defense by having a commons, but also how do you deal with the fact that the attackers can also see the commons and what that might mean for helping people navigate this.
36:29Yeah. Uh, thanks God this is not a question that we have to take from scratch. It has been structuring the entire field of information security for over a decade. And so I think here we can take some inspiration frankly on how things work on the security side. So um if you are doing uh reverse engineering of malware for instance um more often than not you end up using uh open source tools to do the analysis but because everybody uses the same tools you can
36:58share the rule right uh amongst analysts and saying we open source the infrastructure doesn't mean saying you have to share all of the rules for all of the investigative work that you're doing but it does enable uh defenders to collaborate and then go faster together.
37:15And so we see this in Osprey for instance, right? Saying we have a rules engine that everybody can run for free in a way that's permissionless and you can adapt it to your infrastructure doesn't mean you have to publish all the rules uh for how you define and catch spam and define and catch coordinated harassment. But if you do find something that you think is a particularly gnarly activity that infolds on your platform and you have good reason to uh to think that it's also affecting another organization and if they're also using
37:44Osprey, you can share that rule and then you fast forward the collective defense.
37:48And so it's always um that sort of thinking and muscle to develop between what needs to be in the comments to really lift up and enable all the defenders and frankly widen the circle of what is protected online and across AI surfaces versus what is sort of like organizations own things to uh keep and share and you know different organizations will have different preferences on that too which is totally fine.
38:16Actually I have two questions for you.
38:17one coupe, osprey, roost.
38:21What's the deal? What's the deal with birds?
38:23H Yeah, we we like a good pun. Uh uh Eric, you you you would be surprised by how many chicken puns we're able to produce a minute.
38:32So just random. No. Okay. No rhyme or reason. We just like puns.
38:36We just really like puns. And also birds of a feather. Okay.
38:40Exactly. Fly together for safety.
38:42Exactly. Um it's it's sometimes we also repeatedly use the same puns. So now every time I say well you know it's a chicken and egg problem and I uh I will remind everybody that our discord where all those great puns are flying around is completely open and if you want to engage in technical safety discussions while making very fun puns. Uh if you want to be punny uh you can join us on the Discord.
39:07Okay. Right with dad jokes. Great. Glad we solved that. Um second perhaps more importantly uh the conversation is sort of premised. You were saying that the problem with safety nom uh safety startups is you know smite gets acquired and then shut down and so there's a rug pull people who are relying on this but Roost is a nonprofit and so it is also predicated on the idea that people are going to fund it. So one how do you ensure that funding continues and what happens if if Roo's own funding dries up? I feel like you are inviting
39:36me to make a very public call for people to come and fund our work. Please uh please come and fund this work. Uh come and fun this work because it has a very very tangible impact and fund this work too because it's a path to resilience. I love your question because we are building this comce in a way that it can outlast the organization.
39:58Yes. The idea of our work is to bring enough people on board, to bring enough of a community that innovation happens, more people have access to these tools, trust gets built, and if tomorrow we disappear, all of this stays there.
40:13That's the difference between an open- source comments which has this resilience, right? If tomorrow there's no rules, people can absolutely still download and maintain and uh handle Osper and Coup. Now there's a lot of work that we do to again like catalyze this commons make sure there's a common road map. Um I'm particularly excited about some of the scientific efforts we're also building to because now that we have the common road map we we know where they are real gaps and we're able to join forces with institutions like MA
40:42in Quebec or with the UMass Rescue Lab uh to say okay let's let's focus and sort of plug these holes and gaps but but we are building with resilience in mind. I think that's extraordinarily important in the current times, right?
40:58Idea is really not to replicate a hyper centralized system. That's why we chose the open source way.
41:05So, one of the things I I think the resilience that you've described is absolutely fundamental. One of the great things around open source, which is because it's a collective digital commons, it gets all the features of that which is it can be uh it never dies. It can be resumed. It can be steered in different directions.
41:22multiple players can engage in it. It's part of the thing that is magical about open source. So, but there's also a lot of different constituents. So, there's nonprofits, there's universities, there's companies, there's engineers, there's random people.
41:39Say a little bit about kind of how these different elements come together because I think that's important for people understanding the resilience and the vivaciousness and the liveless. And then we're going to start shifting into and how does AI right you know begin to change this the magic of having an open source safety commons is that once the wheel starts turning people start seeing how they can plug in and so you have people
42:09who for instance don't have technical skills but are really really good at building policy and as new big labs are saying oh here's a safety model you can news, but you need to bring your policy.
42:21A few people are saying, "Oh, I can write a very detailed clinically informed description of what um suicidal ideiation looks like in the context of an AI chatbot." And that happens to be super helpful for people who are going to use AI models for this. And I'm going to contribute this policy in open source in a policy pack. Um you of course have great engineers who are coming and are saying, "I see this tool. I have an idea for how to improve it. I'm going to
42:49submit an improvement. Um you have uh scientists who are saying, "Oh, I've always wanted to go and actually test some of these tools and I'm going to go and I'm going to do this and then I'm going to contribute a few insights from fields that have been traditionally left out of how we build safeguards that actually could benefit from this." Um and so the work of an organization like ours is to architect and organize that big collective will around these tools is to uh make sure that everything is
43:18interoperable, right? Otherwise you have like an island of misfit toys with a bunch of things that don't work together, right? What we want is a good strong interoperable comments. We have people using all of our tools together, right? So Matrix for instance is using both coupe and Osprey. Um and then you have people who only use one part of your system, right? So you know the work is to again orchestrate and organize this big collective effort that brings a lot of very diverse stakeholders but that diversity is the strength of an open source commons and I will say in
43:47the team we have amazing expertise from the sort of traditional open source side of things. So I have fantastic CEO and who used to run open source at Google and so she understands how you sort of bring all these pieces together. Cassidy is our absolutely fabulous community manager. is so good at sort of like you know seeing two people go in very different directions and saying actually I know how we bring this into a community uh through office hours we further the project and so yes there is
44:16definitely a muscle uh to orchestrating that big collective and very diverse uh community around it but but this is what gives it the strength and this is what gives us tools that are actually robust and you know better than people doing this alone before we next continue down the AI thing I think one of the things also may be helpful for a lot of people to hear is how does being open source help engage the kind of the intense intellectual and
44:43and scholarly work research work at the university because that's something that most companies don't have any real access to and that's one of the many reinforcement loops that actually provides an advantage here.
44:56Yeah, it's a it's a language translation question, right? So uh if you take this work and publish it with the right colleagues and in the right journals then you know people who come more from the scientific and scholarly practice see it and say oh I understand the language of working through a data set than the paper I can add my piece to this and so again it's about sort of like translating uh that work um uh across different audiences to engage
45:23everybody right now um our director of product Juliet is doing this on the difficult problem of grooming um and thinking about okay what can we learn from law enforcement who's working on this what can we learn from the different really talented academic teams that have been tackling this problem and from the many corporations in the roose community who say we have tackled this we have some context we have some data how do we pull all of these pieces
45:52together that are currently quite separate to really radically accelerate innovation When it comes to grooming detection, we've started this work. Uh, Roblox for instance shared a first um package and classifier that's focused on grooming detection. Got feedback from the RE community, both people who were trying to use it, people who were thinking about grooming, took that feedback and re-released something that's faster, easier to use and loot better. So it's sort of doing this
46:20orchestration of okay let's go and grab these perspective and let's organize some collective push towards very focused discreet well-escribed projects we can tackle.
46:32So coming into the AI age like AI is making it just easier and cheaper for folks to launch apps chat bots everything. Um and sometimes it's small teams sometimes it's one person teams and they could have no background in safety whatsoever. How can these small teams make things safe right from launch?
46:53I have a lot of thoughts on this one. I have personal thoughts, professional thoughts. Um, I suppose I should start by acknowledging that I am myself a compulsive vibe coder.
47:04Great. um vibe coded a number of apps including like family apps and I am excited by the fact that people are able to use AI to bring their own vision uh of something into life faster and faster. Our big vision at Roost is that we need to bridge the gap between how easy it is to come up with a new app, a new service,
47:30and how hard it is to actually do some of the important steps of making it secure.
47:37I think where we feel that the most right now is on child sexual exploitation. We have a lot of builders who are coming in and saying, "We want to do this. we want to do this, right?
47:48And we're very excited when that happens. We're like, okay, we will take you through those steps. But the reality for instance of removing child sexual abuse material from a foundational data set that's then going to be used to train an AI model is that right now it is a hero's journey and you have to go and then register with uh you know the authorities on one hand and then go ask for a hashbank and then go and implement for instance HMA to use this hashbank and then you have to go and ask for a novel CM classifier
48:18and then implement it into something else. And so a lot of our focus is how can we make that process much easier.
48:26The overall vision is that one day it's going to be as easy to launch an amazing website and app than to just type roost at safety.
48:37And then we'll just do it. Yeah. or even you know by default those uh tools will tell you hey uh so fun that you've added a field in your app that allows you to upload say um kids books into an app uh also you may want to run a moderation layer here and here's why and here's how and we'll make it so easy for you. So we want to make that much easier. there's a lot of work to do to do this and we're excited to work again like with others to go and
49:07and tackle that gap.
49:09So um you started with the partnership with open AAI and the GPD OSS safeguard.
49:15Um so what would encourage more Frontier Labs to contribute to Roost? And are the various frontier labs also contributing a bunch of tokens relative to the vibe coding side which is you know kind of the equivalent of like mythos analysis other kinds of things like because that would strike me as a great thing to be doing and I'm hopeful that there's a positive answer here but I figured I'd ask let's make it happen. I think we're also
49:44starting to see consumer demand and the regulatory demand for that which I'm excited about.
49:50And one way to think about it is that's kind of what we expect from other industries.
49:56So for instance, right, if if you wanted to buy a car and I was trying to sell you my car and I would tell you it has airbags. I don't know if they work really well if you're short. Also, it has a seat belt. I think it's made of paper, but I'm not sure. But also, I don't really know what other people use. And you're gonna have such a great time.
50:19You would be like, I want none of this, right? Because in an industry like, you know, cars, we have a set of standards and shared references and shared technology for what we know keeps people safe. um in you know most cars that you can buy these days will use a threepoint seat belt that was developed by Volvo and in 1959 they said we think it's a really good model actually we're going to give all the other manufacturer the right to
50:47do this for free and then we're going to tell people how we do it so that we can continue improving on it right and that's sort of our expectation for how things work that's also what enables innovation right the reason why we have super fun cars that go really fast is because we know how to make them secure and So, I'm also hopeful that there's going to be increasing demand from uh consumers and regulators to sort of tell people like, hey, if you're going around town telling everybody that you feel super satisfied with the way you keep
51:16people safe online and on AI, maybe you should give everybody those tools. Um maybe you can join me in having this public call, right, to all of those amazing AI labs. I know they have more to contribute. Um, and I I can tell you like people are hungry for this tech.
51:34What's really fun when you have an open source stack is you discover people using your tools in production through different and funny ways. So, I'll tell you two stories. Can I do two stories?
51:49Um, there was a small uh fandom uh app.
51:54two people uh they found coupe through LinkedIn actually promised yeah great and they installed it totally on their own we didn't know and they deployed it and in two weeks they found 19 incidents of child sexual exploitation that they were able to report to NECME and then they reached out saying thank you for enabling this work we really wanted to do that we didn't have the tools to do it then we worked with NECME to see okay well what
52:23does it look like to do this experience from the perspective of somebody who has a team of two and I'm and I'm not saying an engineering team of two. I'm not saying a safety team of two people. I'm saying a team of two people. Everything two people and and to think about, okay, how can we make this as easy as possible for builders who want to do the right thing.
52:41That teaches us that the coalition of the willing of people who want to get this right is so much bigger than the people who are currently equipped with the tools to do this. which again is a good news in our ability to quickly make a dent a real dent in protecting people online. Other story actually also LinkedIn related quite fun. Um one day uh the uh lead contributor on coupe is a
53:08formidable engineer uh called Juan receives a a message saying from Atlassian saying hey we're going to submit uh a pull request. can you review uh review it? And we're like, Atlastian's using Coupe. We didn't know that. Uh that's cool. They're big. Uh we're excited that large companies are staking uh you know, some of their uh safety infrastructure on these tools because then it's exactly what's going
53:38on. Then they contribute to it. So it's absolutely awesome to discover the use of your tools by amazing engineering teams knocking on the door saying hey actually yes we've been using it actually we think we can improve it um today if I just think about coupe and ospray I'm not thinking about the uh roost model community and the amazing AI safety models just coupe and ospray the people whom you we know are are using them in prod reaches
54:06around 370 20 million monthly active users across platforms and organization. That's our conservative count.
54:15I think we can do so much more. There are so many more organizations to protect. What's fun about this number is that it's both large platforms and small organizations, two people team or for the first time we also had a journalism team say, "Hey, we're building our own community. Um, we actually want it protected and we're going to use Coupe."
54:37That's awesome. Well, so we've talked a lot, there's a lot more to online safety than keeping kids safe, but we talk a lot about keeping kids safe. And Reed and I also talk a lot about AI companions. And so, getting a little specific, when you're thinking about creating an AI companion for a 15-year-old, what would a genuinely good AI companion look like? And what are the open tools that the developers would need to to build one?
55:03This is a hard question, and this is an urgent question. Um what we know from the usage data of uh AI companions and kids is that um kids are increasingly turning to AI companions and when we look at the types of conversations they're having they're also discussing feelings um and their own problems and seeking support and so there's a lot of really important and consequential
55:31decisions to make science to develop safeguards to build around what does it look like to keep those interactions safe. I am not part of those who think that we should just unplug all the kids.
55:46I we talked about this. I grew up playing with technology. I think it uh gave me a lot of meaningful connections and friends and I think it gave me a meaningful career. And we want kids to be able to play with technology and to engage. We also want them to do so in a way that's safe. I think safe in this context include things like products should be honest about being AI. We haven't really crossed that bridge just
56:15yet. For instance, we I think a lot about AI and toys um today. If you're going to sell a toy that has AI embedded in it, you don't actually need to disclose that. And so you may end up with having an AI toy in your house without anybody sort of thinking twice about it. Mhm.
56:33Uh there I think consensus around companions respecting age appropriate boundaries um and about supporting real world relationships. Right? Ultimately we don't want uh those companions to uh isolate uh young people from their support networks, their friends, their families and their community. I don't think any of that is solved. I think this is a very active area of focus for the field
57:02for roost of course and because this is not solved because this is extraordinarily urgent and because this is hard that is why we're doing this work in the open with scientific institutions like MIA um our uh suicide prevention safeguard we released it in July and what we're saying is we're going to do this work in public and incrementally so that everybody is able to come join in and say okay let's get this right
57:31and we can sort of divide and conquer together to make sure that again we accelerate the pace of innovation because yeah those usages are widespread and I think absolutely we've heard firsthand and in extraordinarily um dramatic ways the cost of getting this wrong.
57:47So it sounds like you don't think we're quite there yet that there is more sort of research and tooling needed before we're able to safely deploy.
57:54Yes. I don't think that we know enough to say we absolutely got this one solved.
58:02I I I am very grateful for the builders who are trying to get this right. I'm even more grateful for the builders and scientists who are trying to get this right in public and with us, but I do think there's a lot of work to do here. And you know, again, I think that the usage data shows that this work is really urgent. Mhm.
58:20One of the classic challenges is if someone's building something trying to include kids, you kind of have two paths given that one is we're just not going to include kids or we're going to include kids and we're going to learn as we go. probably we could only get success by doing the second path. But what would be some of the the oh make sure you're doing this, make sure you're engaging this. Make sure that you're asking these questions so that as you're learning, as you're going, you're contributing back to
58:50everyone else and we're we're making the environment the right kind of safe environment of kids and elevating environment kids that it that they absolutely deserve. And again, I think that there's sort of some of those big questions, right? Does does does everybody know that there is AI involved? Right? Do you have a good sense of what AI is involved? Which safeguards are running around it? Um do you have protections around some areas
59:17like self harm? Um can you ensure uh that uh again you're not building a system that is going to lead to more isolation? These will also depend on the specific types and usage, right? Um, you're going to have somewhat different conversations. Again, if you think about like AI embedded in toys than if you think about uh multimodal AI, uh, where you have a whole lot of different issues
59:45around um, images for instance, right?
59:48Um, and similarly, building for a 12-year-old is a very different story than building for a 17year-old. So I think we need to sort of engage with all of that nuance and and take it with uh utmost serious and do this work rigorously scientifically in public and say this is what we know, this is what we don't know and this is the reality of how those tools are being used and frankly of how kids are being harmed by it today in those very specific ways. So
1:00:18if we brought in from just kids to sort of look at looking at the whole ecosystem and thinking about that chat bots, language models, agents like they're delivering a lot of services that used to be delivered by humans um especially in the safety world. So is there a specific safety capability that you're like that would make the biggest difference if we could bring that online and like how would you show that that works in the real world? Yeah, I mean I have a long list of CCL need and again I think there's a there's a distinction
1:00:47between like problems that we kind of have solves for but they need to be more robust like better distributed uh battle tested invested in versus problems that are a little bit more new that we're trying to wrap our our head around and thinking like oh that is really bad how do we tackle this? I'll give you an example of the letter please.
1:01:08Non-consensual intimate imagery.
1:01:10Today one in 25 American teenager oh god says that they have had a deep fake nude produced of them.
1:01:22I'm not saying one in 25 tell you I know somebody to whom this has happened. One in 25 tell you yes this has happened to me. Mhm.
1:01:31This is an enormous crisis because of course those deep fakes are then used for extortion uh for different types of humiliation, harassment and harm and thinking systematically about okay how do we tackle that novel usage with the widespread um uh scale of of that problem and the very real consequences that requires new creative thinking. uh and frankly like a whole of industry and scientific
1:02:01approach on where do we focus technical defenses and make sure that they're distributed as widely as possible to get ahead of the fact that this is an enormous wave of harm coming for teenagers.
1:02:13I wouldn't have even thought of that one and now it's going to haunt my dreams.
1:02:18Well, that's why we have all the puns is because let's get it solved.
1:02:21Let's get it solved. Yes. So I mean as part of that um so how are the collaborations with the frontier labs and other AI labs um what kinds of things need to improve?
1:02:33What are the ways that government civil society can help this as well? like how do we get that in a faster and stronger kind of thing because for example solving the the you know whatever I don't know what the exact term for the uh sexual abuse from deep fakes is but like solving that is critical and the kid doesn't even have to be a user of it for that right it's like that's
1:03:01the it's not even like yes how they use it it's other people using this so what's currently going on and what should we be doing to make it better.
1:03:09I think this is a a call for everybody to roll up their sleeves and work together. Um, we need more attention to this problem. There's a lot of really good civil society groups who are doing these surveys. That's why we know that this is happening and who give us a lot of context on okay well what does that look like for a kid? How does that happen? Um there are a lot of players uh who have technology to contribute that can help mitigate this harm at sort of
1:03:38the different areas where it needs to be stopped, right? Um and we want that technology to be contributed in the open commons. Um we also do need a philanthropic organization to say we're going to take a bet on the fact that we can accelerate this work. We can do this in public. We can do this with the rest of the scientific community. And and as I said, I think that there are a lot of expertise that live in civil society and in the scientific community that for too long we've said, I know you don't belong in the safety and safeguard
1:04:08If we want your input, you will sign a stack of NDAs. You will come to uh some uh office in Silicon Valley. We will hear you and then you will leave and you will never know what we took from your remarks. That's not a good model. That is not going to give us the pace and scale. um an excellence that we need in the face of these problems. So I think this is this is one where everybody can really uh uh bring something. I'm somewhat encouraged and also somewhat
1:04:38disturbed by all the conversations that we're having right now on pacing the frontier and how everybody needs to bend together to do something. There is also subfrontier problems that are very urgent, very catastrophic and that we absolutely should bend together to focus and solve these problems.
1:04:54Right. Right. And I mean and so much of the conversation is also about global cooperation and what is happening around around the world. And so you were recently speaking at the G7 in France and you were talking about the importance of open source, digital sovereignty, child safety. Can you talk about why this is so important in the context of European builders and so like why is it helpful especially in that context?
1:05:17Uh yes. uh I was invited to contribute to the uh G7 and not only because of uh some of my um academic work with a broader coalition of uh colleagues on defining what we mean when we say open source AI because of course open source has had a traditional definition uh that we've worked through a lot for software but when it comes to AI it's a little bit different and in fact there are different ways to
1:05:45do open source when it comes to AI and Open- source AI can itself take different shapes. It can be just open weights or it can be everything is open including documentation and data sets and safeguards or it could be many different combinations of this is open but not that and you sort of have to think about the entire stack and for each component of the stack you have to think about what would it mean to open that and then you think about like what am I trying to achieve and
1:06:13which shape of openness gets me there. That conversation is very relevant to sovereignty because of course um you have the European Union but also most recently Canada with their big AI strategy saying well we want to be able to own uh not rent the different components of the stack that matter to us. uh that is also extremely relevant for safeguards, right? Because similarly uh if you are going to legislate on
1:06:46child protection, you also want to make sure that people your own ecosystem of innovators have the means to comply with your law. Otherwise, the more we call for regulation, more rules on this topic without investing on open- source tools, the more what you're actually doing is further concentrating power in the industry and closing down the market around a handful of incumbents who have the safety engineering tools that are and and teams that are able to comply
1:07:15with those laws. Right? So, it's critical to your ability to protect your citizen uh and to make regulation in this area. And it's also critical to your ability to have um you know independent and and strategic and sovereign uh uh versions of your stacks.
1:07:34Well, going a little bit more global, one of the challenges when we deal with a lot of the prospective harms is that for example, you know, abuse of images can be created anywhere. Yeah.
1:07:44Um and so and distribution networks and all the rest. So what needs to happen globally, right? Because you know one of the questions is is there will be image generators for example running in the vast majority of countries there's already open source versions of them or open weights versions of them and there's nuances you were gesturing at there so what what needs what should be thought of as some good basics or some
1:08:12good starts for the G7 coming together for for the UN we're here in UN General Assembly week you know what what would be the kinds of things that would be good things to do globally.
1:08:25I think my first ask is never use the idea that you can't root out all of the problem to justify not taking the extraordinarily important incremental steps that really are going to save lives and reduce harm.
1:08:42Yes, there will be people doing this with models that we don't have control of in places that we won't see. But also, when you look at the way the majority of these harms are being perpetrated, it is happening with apps that are being advertised on social media, distributed through app stores.
1:09:01uh and in context where there's actually so many different levers that we can identify and say here there's a good chunk of the harm that if we focus and join forces we can remove. So I'll say that's my sort of first most important goal which is like let's get progress done. Then we'll worry about the sort of remaining part that we can get to. But there's so many lives we can save harm we can remove uh good we can do by focusing on all these incremental steps
1:09:30that we're right now not investing enough on. You've described your generation as internet optimists who believe cyber space would spread democracy and expose corruption by default. We all heard this. We were so excited and they came true. There have been uh you know many good things that technology, social media, the internet etc. have have brought about. Um we didn't though predict all the harms and you know we watch some of these beliefs get weaponized and so is the optimism
1:09:59you still carry. Um, is that a conclusion from the evidence that you've seen or you are just an optimistic person? Like how would you say your optimism optimism has sort of uh waned and waxed over the years? There is somebody somewhere who could probably answer the question of what is optimism?
1:10:18Where does it really come from? And I would like to know um I think maybe in my case perhaps it's a personality trait. Um I am part of this generation who is extraordinarily excited about the power of um technology and internet. Um I remain extraordinarily excited about it.
1:10:37Uh over the years I've had to work on violent extremism, terrorists use of these systems, uh foreign interference, uh hate mobs, uh suicidal ideiation, child sexual exploitation. And so my colleagues often say that I am maybe the most optimistic person looking at the darkest corners of our technological life the worst.
1:11:00Um but I always see those reasons for optimism. The first one is that there is so much we can do if we join forces. We know what to do. We just need to focus and we are still in an environment where we can make very tractable progress by doing things that we know how to do. And so that gives me a lot of optimism because uh then I know where to focus my action. Right? There's this question of like does action come first or does hope
1:11:29come first? I actually think that hopes comes from the action. Um the other thing is I am very excited by the next generation of builders. Um, sometimes when we think about child safety, we can describe the next generation as sort of a big unified mass of youngsters to which tech is happening.
1:11:52But that's not the case. There are also the builders of tomorrow. There are also very creative and fun and there are a bunch of maker pirates within them too.
1:12:02And it's fun to see what they do. Um there's a a hashtag online that says cyber deck for instance where you can see uh young girls take poly pockets and inside they build mini computers and then they run local AI models. That's really cool. The next generation is building systems that they want. They're going to build the technological environment that they want to see. And I think we need to bring them at the table, not just as victims, but also as builders and shapers of our
1:12:31technological futures and everything that we do at Roose, we also do for them. And it's super uh optimism producing to see them knock at the door and say, "Hey, I'm going to build this uh weird app. I need all these safety tools to do it." And we're like, "Yeah, go go go. You don't need our permission.
1:12:46Go to your app." So, yeah, that's exciting and that produces optimism. So, one of um the kind of phrases around this that I really like is cyber piece.
1:12:55And it's it describes not just fewer bad outcomes, but actually a kind of a healthy and dynamic and growing system kind of an equilibrium. And you've talked about it some today. um with pluralist audible safety, you know, kind of ecosystems, communities get more uh nuanced and detailed and specific to their community kind of agency. Um
1:13:23what do you think are the most important things to help continue accelerate to that path? Like what's what what should people understand about what kind of ecosystem we're trying to get to?
1:13:36Why should it be kind of almost like a what? You're not using these tools, you're not participating in this that why not, right? You know, it's kind of as as the why the why not. What what do you think it takes to get there?
1:13:49So, I've done my um academic research and ultimately my dissertation on this concept of cyber peace and that is because I was um amongst a group um who was studying cyber warfare. Um I had two absolutely fantastic adviserss who encouraged me and supported me to sort of look at the problem maybe from a different angle. Um and and and I was taken by the fact that everybody was very focused on defining cyber war, defining the threshold of armed
1:14:17conflict. And it's as I said not not unlike the conversations we're having right now with the hyperfocus on the frontier. But there is all this great theory of positive peace building. What happens below the threshold? If we're so intent on defining cyber war, do we know how to define cyber peace? Do we know how to invest in the mechanisms that make our online lives livable? Uh that make room for thriving, that make room for cooperation. Uh I think that the
1:14:45disappointing answer to your question is that some of that is the power of our focus. We get to decide where we want to spend our energy, where we want to spend uh our efforts, where we want to build, where we want to spend. And saying we care about making the subfrontier, subwo space livable, we care about tackling these harms that are immediately in front of us is a decision we can make um with our focus and and making sure that
1:15:13we can keep centering these in the conversations we're having around AI, around harms, around AI safety. I think is part of the problem. We don't need to have this um opposition between catastrophic risk and immediate risk. I don't think that opposition is particularly I of course I understand why it exists, but we we don't need that strong of opposition. We can make sure that we focus on both sides of the line.
1:15:40Um and that we do both at once.
1:15:42This is just reminding me it's like government works well when you forget that it exists. It's like none of us know that we're getting tons of spam text messages a day because that system actually just works and some of the spam texts are just filtered out. We still get some but most are filtered out. Like all of these systems when they just work, you in a good way get to take them for granted. It's sort of like the Maslo's hierarchy. It's like if safety is the floor that's like we have to have it exist before we can get to flourishing and before all these great
1:16:12things can happen. And so what does that actually look like once we have safety as the floor? Yes. Yes. And then what does that look like when cyber piece is real for a teenager, a platform, government, like how to paint that picture?
1:16:24I I love uh that description like yes, we want to become ubiquitous and you don't think about it.
1:16:30Totally. And things are just more safe and that is just the way it happens to be. Um and frankly, we have that on some corners of our online lives. How often do you think about the amazing work that Les Encrypt is doing?
1:16:43I don't know. Maybe maybe you do. But but generally when people start a new website, they're not going to their office thinking a I need to figure out cryptography, right? Like no, we don't, right? We we've we've managed to sort of like have building blocks and institutions that raise the floor for everybody. I think we can do this for safety. I think it will look different in different products because again, we don't know what people are going to build. What we know is there are
1:17:11fundamental things that need to be done in order for those users to be safe. Uh that some of this we know how to tackle and we need to make those technologies available to everybody. Some of this we don't know how to tackle yet. We need to focus make that scientific effort, build those new safeguards and then again will this look like uh new very fun chatbot interactions maybe. Will this look like less spam? Hopefully uh less fraud uh please that would be great. Uh less
1:17:39child sexual exploitation. Absolutely. Uh less uh interactions with chatbot that leads to uh violent extremism. Yes, please. All those things, right? But like yes, the idea is exactly what you said, which is let's make it ubiquitous so that it's it's much easier to build from a shared safe foundation.
1:18:01All right. So, let's go to rapid fire and you can answer at any length. These are just the same questions we ask all of our delightful guests. So, is there a movie, song, or book that fills you with optimism for the future?
1:18:15Uh, many. I will pick one because I think that's the game.
1:18:20So, if you want to do two, it's fine.
1:18:24I love Star Trek. I am a Treky of the Deep Space 9 category. And I'm not sure why, but it actually turns out that a few people around Roost and in the Roost community also are Trekies. So I think that there is something about Trek uh inciting optimism about the future, optimism about our ability to collectively govern it at interplanetary scale. Um but yes when I think about our
1:18:52board for instance we have um amazing people like Audrey Tang or Megan Smith or Maria Resa and all those people are also Trekies and so it has happened uh that we have very serious roose discussions and then the end is like okay okay good live and prosper.
1:19:10Awesome. What is a question that you wish people would ask you more often? I wish people would ask uh who should be at the table when we make safeguards. uh again I think we've inherited bad habits of doing this a little bit uh on our own in the dark in a way that's uh very opaque and and sort of siloed and there is another way to build and thinking about well who else has interesting expertise to contribute on a very difficult societal problem is a question
1:19:39I wish people would ask more and I think we will bring more people and uh yeah more perspective so where do you see progress or momentum outside of call it you know all the things we've been talking about your industry but uh like another area that you would like to shine a light on that inspires you.
1:19:59Well, it's it's interesting. I know you said another area, but for people who've been working on child safety, we are going through a very interesting moment where that conversation is becoming a kitchen table conversation. And so I am quite inspired uh and energized by the fact that many people around us are starting to think about okay well what type of relationship do I want between young people in tech and what is my
1:20:27piece of it. So I'll give you a very bizarre example. I recently became a mother and uh as a result my uh daughter um goes to daycare and they had a conversation on well we have Amazon Alexa in the classroom but the wake word that we use is Alexa and in an environment in which this generation is going to grow up with companions is that what we want because that's anthropomorphizing the relationship to
1:20:56the computer. Now, it turns out, here's a Trekky secret that one of the other few available default that you can use for the wake word is computer.
1:21:08Like in Star Trek. And they really didn't know that. That's cool.
1:21:12You can go and change it tonight. And so they were thinking about how to do their part in finding the right relationship between youngsters and machines as everybody sort of grows up again surrounded by AI. uh and changed the wake word um from Alexa to the very trekky computer. And I'm excited by the fact that many people are having this conversation at so many different levels, right? In classrooms, in boardrooms, uh and this is becoming much
1:21:42more of a mainstream conversation.
1:21:44In my house, my kids just try to say, "Hey, Google," in the craziest accents, at the loudest volumes, at the strangest tenor. So, we we haven't quite No, I That's great. We're not at that stage yet, but this is super. I think you have red teamers on your hands. I think you should send them to Bruce to go and red team some safeguards.
1:22:03They're very into it. Uh, all right. Our final question. So, can you leave us with a final thought on what you think is possible to achieve if everything breaks humanity's way in the next 15 years? And what's the first step to get there?
1:22:15Ubiquitous safety for everybody. Come roll up your sleeve and build it with us. We'll we'll get there one step at a time together.
1:22:25Karine, thank you so much.
1:22:27So live long and prosper.
1:22:31Possible is produced by Pallet Media.
1:22:33It's hosted by Araf Finger and me Reed Hoffman. Our showrunner is Sha Young. Possible is produced by Tanasi Doss, Katie Sanders, Spencer Stramore, Emozu, Amansuri, Danny Garrison, Trent Barbosza, and Tafadisa Neimarunway. Special thanks to Surya Yalamanchilli, Sida Sabieva, Ian Alice, Greg Biato, Parth Patil, and Ben Relis. And a big thanks to Jeremy Ponac, Anne Bertusio Dehalbeck, Jacqueline Ortiz Ramsey, Danielle Hogan, and the team at Lighthouse
