Discover
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)Assessing the Risks of Open AI Models with Sayash Kapoor - #675
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)Assessing the Risks of Open AI Models with Sayash Kapoor - #675
Assessing the Risks of Open AI Models with Sayash Kapoor - #675
Update: 2024-03-11
Share
Description
Today we’re joined by Sayash Kapoor, a Ph.D. student in the Department of Computer Science at Princeton University. Sayash walks us through his paper: "On the Societal Impact of Open Foundation Models.” We dig into the controversy around AI safety, the risks and benefits of releasing open model weights, and how we can establish common ground for assessing the threats posed by AI. We discuss the application of the framework presented in the paper to specific risks, such as the biosecurity risk of open LLMs, as well as the growing problem of "Non Consensual Intimate Imagery" using open diffusion models.
The complete show notes for this episode can be found at twimlai.com/go/675.
Comments
Top Podcasts
The Best New Comedy Podcast Right Now – June 2024The Best News Podcast Right Now – June 2024The Best New Business Podcast Right Now – June 2024The Best New Sports Podcast Right Now – June 2024The Best New True Crime Podcast Right Now – June 2024The Best New Joe Rogan Experience Podcast Right Now – June 20The Best New Dan Bongino Show Podcast Right Now – June 20The Best New Mark Levin Podcast – June 2024
In Channel
Download from Google Play
Download from App Store
United States00:00
00:00
1.0x
0.5x
0.8x
1.0x
1.25x
1.5x
2.0x
3.0x
Sleep Timer
Off
End of Episode
5 Minutes
10 Minutes
15 Minutes
30 Minutes
45 Minutes
60 Minutes
120 Minutes
Transcript
00:00:00
[MUSIC]
00:00:03
>> All right, everyone.
00:00:04
Welcome to another episode of the Twimal AI podcast.
00:00:07
I am your host, Sam Charrington.
00:00:09
Today, I'm joined by Sayesh Kapoor.
00:00:11
Sayesh is a PhD candidate in the Department of Computer Science, as well as a researcher in the Center for Information Technology Policy at Princeton University.
00:00:20
Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show.
00:00:26
Sayesh, welcome to the podcast.
00:00:28
>> Thank you so much for having me.
00:00:29
I'm really excited to dig into our conversation.
00:00:32
We'll be talking about a paper you recently published on the societal impact of open foundation models.
00:00:39
Before we do, I'd love to have you share a little bit about your background and how you came to work in the field.
00:00:45
>> My background is in computer science, and I started off my research career working in theoretical machine learning.
00:00:51
From there, I started quickly pivoted to investigating fairness in machine learning algorithms, and then more broadly to the societal impact of AI.
00:00:59
Over the course of this research career, I've worked at IIT-Confer for my undergrad, then did research in said Columbia University and EPFL in Switzerland.
00:01:09
I worked at Metaford two years in their community integrity team, building machine learning in order to code some of the harms that you're talking about, and then I began my PhD at Princeton University.
00:01:18
>> With regards to this paper, it's interesting timing, I think at least in the context of the podcast, we recently published an interview on ALMO, which as you well know and mentioned in the paper is an open model that is really seeking to promote openness strongly,
00:01:36
and the paper also reminds me that when GPT-2 came out, we held a big debate as a webcast on whether open AI should make this available,
00:01:49
should it be closed and should there be very limited access to it.
00:01:54
We've since come a long way and models much more powerful than GPT-2 are readily available.
00:02:00
What this paper seeks to do is to really ask the question, or a set of questions around that fundamental openness that is to a significant degree open to debate.
00:02:14
I'd love to start by having you share a little bit about your motivations for the paper and how it came about.
00:02:21
I guess the main impetus for the paper was we were in all of these conversations where we were talking about the benefits of openness on the one hand, and then in other separate conversations when we were having into the risks of openness.
00:02:34
In particular, there was a lot of attention to papers that claim these catastrophic risks from language models.
00:02:40
There were a couple of studies that came out of MIT last year that pointed to the bio-security risks that having access to an open language model could allow malicious users to create bio-weapons.
00:02:51
In the midst of all of these conversations, one thing that stood out was just the lack of a common ground when it came to even talking about these risks.
00:02:59
What is this risk we're talking about?
00:03:02
What is it being compared to?
00:03:03
Is it compared to the risk of someone finding out how to create a bio-weapon on the internet?
00:03:08
If so, how do we go about conducting these risk assessments?
00:03:12
In this almost fractured debate, all of us, like me and Rishi and some of the other authors, felt there was necessity to come up with a framework that first removes a lot of the misconceptions about openness and what risks it enables.
00:03:26
And second, it also opens the floor for more constructive debate going forward on what the risks are and how we should band together and collectively address these risks.
00:03:35
One of the things that struck me about the paper is the diversity of authors that are named in the paper.
00:03:44
Several folks have been interviewed on the podcast before from a variety of organizations, academic and non-academic.
00:03:52
I'd love to have you share a little bit about how that came about.
00:03:56
Absolutely.
00:03:57
I think the origin story of the paper is that last September, we organized a workshop called The Responsible and Open Foundation Mario Workshop.
00:04:06
So this was a giant effort by Princeton and Stanford.
00:04:09
And as a part of that, we really brought together like a wide range of perspectives, but all of these had one thing in common that taught deeply about the impact of openness on society.
00:04:18
So we looked at perspectives from the industry, from academia, from civil society organizations.
00:04:23
And following that workshop, it became clear that there is a need to write something more cohesive about the impacts of openness.
00:04:32
And it really became clear that a lot of the evidence that we were talking about was not very well-sustantiated or a lot of the claims did not really have the bar for empirical evidence that we would have liked.
00:04:45
And so really, like in some sense, the papers follow up to that workshop wherein a lot of these experts came together and tried to set some common ground for how we should discuss the merits and benefits and risks of openness.
00:04:58
And one sort of other type of organization that's sort of referenced in the author list is policy makers.
00:05:04
So I think people who are trying to influence policy or influence how policy around open foundation model plays out in the real world.
00:05:13
And that's because apart from all of the debate that's happening within the AI community or within the research community, policy makers are also very actively considering the question of openness.
00:05:23
So we have ongoing policy proposals in the US, the UK and the EU, specifically looking at the impact of open foundation models.
00:05:31
In some cases, it's to impose stricter barriers such as in the White House executive order.
00:05:36
In others, it is to provide carveouts, for example, in the EU.
00:05:39
But I think having this framework to assess the societal impact of foundation models can really help these policy cognizations as well.
00:05:47
So many of the people who wrote the paper with us, we were very happy to have the policy expertise on board.
00:05:52
One of the very first things you do in the paper is define what you mean by an open foundation model and you define that in terms of the weights being freely available.
00:06:02
I'm curious the degree to which you think that very definition is contentious or has nuances or is worthy of discussion in and of itself.
00:06:13
Absolutely.
00:06:13
I think the definition of what constitutes open foundation models are even open source foundation models.
00:06:19
I think it's one of the most contentious debates that going with going on today.
00:06:23
The reason we chose the specific definition and the words we did was first, we did not want to call these models open source foundation models because open source implies something completely different about an artifact.
00:06:35
It means that the code and to some extent the data used to train the model is made available.
00:06:39
You have the documentation to reproduce the data.
00:06:42
There is no use restrictions on the model at all.
00:06:45
So that's why we deliberately chose not to use the word open source but rather just open foundation models.
00:06:51
A second part of it was you might have seen the author list going back to that for a second.
00:06:55
The executive director of the open source initiative Stephano Maffuli is also one of the authors.
00:07:01
He is actually leading this charge of the open source initiative defining what open source really means for AI.
00:07:08
So to some extent the term open source AI is not even that defined right now because the open source initiative does not really have a definition of that.
00:07:15
So both of those meant that open source is out of the window.
00:07:19
Now when it comes to the question of the risks and benefits of models, we chose to focus on model weights because that's where a lot of the perpetrated risks of these models come from.
00:07:29
So for many of the benefits of openness, you might need access to the code and the data and projects like Oelmo and Pitia from Illuter AI are like stand out examples of how far you can take openness with checkpoints and model weights and data.
00:07:44
I think for a lot of the risks, all you need is access to the model weights alone.
00:07:50
And I think this also comes through in last year's executive order from the White House which focuses on foundation models with widely available model weights.
00:07:58
So I think we really wanted to hone in on this specific question because once the model weights are released to some extent this decision is irreversible and this fact about releasing the model weights openly has caused a lot of concern about releasing open foundation models.
00:08:14
And so that's why we chose to stick with foundation model weights that are widely available and with the term open foundation models but not open source foundation models.
00:08:23
Many of the concerns that are referenced in the paper are still concerns in a world where the weights aren't released but you do reference the idea that with the models behind an API of sorts there can be monitoring there can be services can shut down particular uses that are abusive.
00:08:44
And to some degree you kind of call out that once the weights are out it's a bit of a Pandora's Box being opened whereas behind a service there's still some optionality.
00:08:56
It was that a core idea in the way you thought about this idea of openness.
00:08:59
So I think again like going back to the concerns that people have had with openness a lot of these are tied to the fact that once foundation model weights are out there basically anyone who can download these models of the internet can do with them whatever they please.
00:09:14
So there are some mechanisms you might envision for reducing the harmful impacts.
00:09:19
So for instance if a model has been known to cause harm you might get it removed off of hugging face and GitHub so that the model weights aren't hosted.
00:09:27
But to some extent a lot of these interventions can be easily circumvented for instance through a model weights being uploaded to target websites.
00:09:35
One of the reasons we focused on the concept of model weights was precisely because the risks around models are at least theoretically amplified when you have no takebacks when you cannot take back the model weights or control who uses them or for what purposes.
00:09:50
One of the things that is very clear when you read the paper is that you're really trying to create a framework for folks to communicate about these various risks and you get the picture very quickly that you've been in conversations where people are talking about risks but they're really talking about very different types of risks and they aren't able to reconcile how to bring them all together.
00:10:17
How did you conceive of a way to reconcile that?
00:10:22
So I guess one of the things that was most helpful was seeing how two papers that came out of MIT last year dealt with concerns around biosecurity risks.
00:10:32
In a follow up work though I think a couple of months after this paper was released group of researchers from Stanford looked at what this information was that the authors were arguing would lead to future pandemics being caused by language models and it turns out that all of the information that was at stake here was easily available on Wikipedia.
00:10:53
We can have a lot of harmful outputs from open language models but if you can get the same exact information from web search on the internet perhaps looking at curbing the release of open foundation models is not the right policy proposal.
00:11:06
So this really informed us on the need for creating this common ground because if you don't have this common ground it'll essentially be researchers on the one hand pointing out all of the risks of AI and on the other hand pointing out how they're just equivalent to past releases or basically equivalent to information widely available on the internet.
00:11:24
The results of that second paper are fairly intuitive to anyone who's familiar with language models and the idea that to some degree they are pulling together information that's already in their training datasets.
00:11:38
Why do you think there was so much contention around that idea?
00:11:42
So I think one of the reasons was that for a long time language models just did not work very well.
00:11:48
So if in 2015 someone had come up and said you know language models can help someone cause biosecurity issues I think that concern would not be taken seriously at all because the state of progress of language modeling was such that the state of the art language models essentially could not offer any help.
00:12:04
I think that has sort of changed in the last five years or so and especially with instruction tuned models it has become easier for people to rely on language models for assistance.
00:12:14
To be clear I'm not saying that the papers results are flawed or anything I think it's important empirical evidence it's important to know that we can use language models to extract information about biosecurity risks and it's very important to know what the capabilities of these models are.
00:12:31
But at the same time our main point is that we shouldn't be caught up by focusing on just the part where we have a language model or access to a language model alone.
00:12:40
So for example there is a follow up study by Stephanie Batallis in foreign policy where she points out all of the things someone might wanted to in order to create a bio weapon and essentially finding out information about the bio weapon is a very small part of this pipeline.
00:12:56
A lot of this information is available on the internet.
00:12:59
It's available in AP bio courses and so if our aim is to look back and curb biosecurity risks then perhaps looking at the language model is not the biggest joke point that we have.
00:13:11
There are other things that we can do for instance imposing bands on DNA synthesis or more restrictions on genetic screening and so on.
00:13:18
And so our risk assessment framework is essentially trying to open up this space of risks that have been called about because our call to have been coming out of language models and opening it up to see where the most useful joke points might be.
00:13:33
One of the big contributions of the paper is defining the risk space in terms of marginal risk.
00:13:41
Can you dig into that a little bit more?
00:13:42
Absolutely.
00:13:44
So marginal risk really means what is the risk of language models or foundation models more generally compared to previous technologies.
00:13:53
So when focusing specifically on open foundation models we think there are like two things against which we should calculate marginal risk.
00:13:59
One is the existing state of the affairs with existing technologies like the internet and the other is foundation models that are not released openly because a lot of the policy efforts at regulating foundation models are specific to open foundation models.
00:14:14
So the second sort of comparator is what if the foundation model was not released openly what if it is a closed foundation model.
00:14:20
And to come up with the framework for assessing marginal risk we bank on the cyber security threat modeling framework.
00:14:27
So threat modeling is a concept in cyber security which basically allows cyber security researchers to come up with the entire pipeline of how a risk actually materializes in the real world.
00:14:38
It has several steps like reconcise or like figuring out what the systems are what the loopholes are and essentially coming up with like an entire framework of how to do this in a repeatable way.
00:14:49
So we take inspiration from this framework and we create this six step risk assessment framework for identifying the marginal risks of open foundation models.
00:14:57
The first step is threat identification.
00:15:00
So it's important to know what specific threat we are looking out for and who this threat is from.
00:15:05
So this latter point is important because it's very different if a threat is from like let's say a few individuals or a small organization versus from state-backed actors.
00:15:15
They have very different levels of resources so it's important to understand what this threat is where this threat is coming from.
00:15:21
The next two steps are about the existing level of the risk because in many cases the risk we're talking about also exist in the real world as well as the existing defenses that we have against those risks.
00:15:31
So taking together these three points allow us to then think about the marginal risk of releasing foundation models openly.
00:15:39
So this is compared to existing risks for instance from web search on the internet or closed foundation models as well as how open language models or open foundation models might allow us to supersede the existing defenses that we have.
00:15:54
And similarly like once we've sort of come up with the marginal risk of releasing open foundation models it's also important to look at how easily we can defend against this marginal risk because in some cases while open foundation models might allow new risks to materialize they might also be useful for defense.
00:16:10
And then finally our last step in the framework is very precisely stating what the uncertainty and the assumptions in this entire analysis are because in many cases it's these uncertainties and assumptions that lead to the most prevalent points of contention between people arguing on both sides of the open versus closed debate.
00:16:31
I'm wondering with that framework defined if you could walk us through an example whether it's the bioterism or another example of how you would apply the framework step by step.
00:16:44
So in the paper we carry out this analysis for two areas.
00:16:48
The first is cybersecurity risks and the other is the risk of non-concentral intimate imagery such as technologies like deepfakes.
00:16:56
So maybe I can talk through the latter example which is non-concentral intimate imagery or NCII.
00:17:01
So first I mean the first step of the framework is threat identification where identify who the threat is from.
00:17:08
In many cases we've seen that the threat of NCII is actually from individuals or small organizations.
00:17:14
So this might be someone you know who's creating deepfakes.
00:17:18
In some cases it is deepfakes of celebrities and others it is people they know in real life.
00:17:22
And this threat I mean in terms of the existing risk this threat has been around for a while.
00:17:28
So we've seen many types of digitally altered NCII images being created through tools like Photoshop for example.
00:17:35
And similarly in terms of existing defenses there are few defenses that people have.
00:17:40
The first is social media platforms like Facebook and Instagram and YouTube have channels to report NCII if someone posts your image you can report it and it'll be reviewed.
00:17:49
Similarly there are some federal statutes against the sharing of non-concentral intimate imagery both in the US as well as outside like in the UK and the EU.
00:17:58
So this brings us to the question of marginal risk.
00:18:00
In this framework or in this current scenario where we have Photoshop and people do share NCII of other people at some rate of prevalence what is the marginal impact of open foundation models.
00:18:12
So for this specific risk we find that the marginal risk of open foundation models is actually pretty high.
00:18:17
Several analysis have shown that since foundation models like stable diffusion have been released openly the amount of NCII prevalent on online websites has gone up dramatically.
00:18:29
Compared to tools like Photoshop the use of this model might not require any expertise in digital technology at all.
00:18:37
And compared to earlier enforcement strategies like going on social media and reporting your image I think advocating for taking down AI generated NCII is a little bit harder simply because of the legal status of AI generated images right now.
00:18:51
This is an interesting sort of policy rabbit hole but back in the 90s the supreme court ruled that virtual pornography is protected under the first amendment.
00:19:00
People have a first amendment right to create and share these pictures even if social media platforms later take them down that's their prerogative under section 230.
00:19:07
Like just the creation of these deepfakes it's as of yet still a contagious first amendment issue and it's unclear if the use of foundation models for generating more believable or more realistic eventually will sort of cause the court to change its opinions.
00:19:20
I think we have to that's a tremendous example of legal frameworks not keeping up with technology.
00:19:25
Exactly and in some sense the supreme court verdict in the 90s case was actually it had great foresight because at the time the state of virtual pornography was such that no one would mistake a virtually generated image from a real image of a person but now when we have foundation models like generative AI models widely available and easily downloadable you like I can run civil diffusion on my MacBook it just becomes really hard it's like a difference in kind not just a difference in quantity and so I'm sure we're likely to hear many cases on virtual NCI and generative AI very soon.
00:20:02
You've mentioned factors including the ease of creation the prevalence or the increase in occurrences it sounds like many of these factors are very much themselves kind of upper debate or discussion and it's ultimately a judgment call on behalf of the person who's making the argument but what you're trying to do is ground it in at least we're going to talk about this factor is that a fair assessment.
00:20:28
That's absolutely right and that's why the last step of the jiske assessment framework is specifically about the assumptions built into the risk assessment framework throughout like whether you think these models will continue to improve at the same rate or whether we're seeing like somewhat of a plateau there are also assumptions about the quality of the generated image and so on and so I think that's the step where you sort of ground all of your subjective calls in this risk assessment framework and explicitly call them out.
00:20:55
The final point of the marginal risk assessment is comparison to closed foundation models and here too at least based on the empirical evidence we've seen so far close model providers like OpenAI have been quite successful with their guardrails.
00:21:08
There was this incident involving Kayla Swift's NCI that was actually using Microsoft's tool which is a closed model but at the same time what this allowed Microsoft to do is very quickly patch their model so that it wouldn't generate NCI of real people and this is just not something that's available as a redressal mechanism to developers of OpenFoundation models.
00:21:29
In general I think it's clear that for NCI I at least the marginal risk of OpenFoundation models is pretty high and when it comes to the ease of defenses I think this allows us to look at this whole pipeline of the creation of NCI and come up with a few spots where we can reduce the harm.
00:21:47
So the pipeline for creating generative AI waste NCI is you have a model you might have platforms where those models are hosted you might have downstream platforms like social media where people might upload these images into mentions that the model level are quite hard it's very hard to like eradicate the spread of stable diffusion simply because you can download this file is like a four gigabit file you can run it locally on your MacBook you can also run it on an iPhone actually and so it's really hard to track the usage of these models but when we come to the next two sort of layers of the pipeline interventions become somewhat easier so one example for the middle layer platforms where models are shared is this startup called Civitei.
00:22:30
Civitei is a platform for people to share foundation models that can generate images of various sorts one of the uses of Civitei in the past has been to post bounties for models that generate NCI about specific people.
00:22:45
So this example is basically like one where tangible harm is being caused because an online platform is failing to take down models that can cause harm in the real world.
00:22:56
A few weeks after this very nice investigation from a news outlet called 404 media reveals this fact Civitei has imposed stricter guardrails on how these models can be distributed and similarly I think for social media platforms as well so I think social media platforms need to do a much better job of curbing down on clamping down on NCI and one way they can do this is by allowing users to take some of the power back in the hand so there is this nonprofit organization called stop NCIi.org where if you fear that one of your intimate images is about to be shared on the internet or perhaps has already been shared you can upload a hash of that image to stop NCIi.org and when this image is uploaded to any social media platform that coordinates with stop NCIi and I think as of today this includes Facebook,
00:23:45
Instagram, Reddit and so on.
00:23:47
Any of these platforms if that image is later uploaded that image will be automatically and proactively removed so that people don't get a chance to see it.
00:23:55
So I think interventions like these are much more tenable compared to interventions at the model level simply because of the high costs of non-proliferation of of these models.
00:24:05
So this is what like the risk assessment framework and especially the ease of defense step in the risk assessment framework tells us like where we should focus our defenses but at the same time some of the harms would likely still continue so even though public platforms and AI model hosts might take down these models you still have end to end encrypted chats and you have telegram bots that create NCIi of people.
00:24:28
So to some extent this marginal risk of open-pondition models for creating non-consensual intimate imagery is tangible and it has already caused your world harm and then when we come to the last step which is the uncertainty and assumptions I guess some of the assumptions are around the legal state of affairs so in particular that end to end encrypted chats would still continue.
00:24:50
I think there is a couple of initiatives especially in the UK which are trying to undo end to end encryption in the first place.
00:24:58
I think that's a very dangerous and risky policy proposal but nonetheless it is one of the assumptions built into our analysis.
00:25:05
Similarly the assumption around these models already being useful for creating NCIi means that even if like future models can have guardries current models will already always exist and they can always be used to create NCIi.
00:25:20
So the second this assumption says that even if we can somehow curb the harms of NCIi in future models that doesn't really matter if current models are already good enough to have this huge much in a risk.
00:25:33
You referenced in your explanation the overwhelming benefit of proliferation of open foundation models is that part of the analysis is that your opinion is that something that would end up in the assumptions phase of an analysis.
00:25:51
So the risk analysis framework does not take into current the benefits.
00:25:54
It's not meant to be like a cost-benefit analysis.
00:25:57
I think the cost-benefit analysis is hard to do in like a generalizable way simply because different organizations might have very different incentives they might view different benefits differently.
00:26:08
So in order to scope the framework to something that was like generalizable but also specific enough to be useful I think we focus specifically only on the risks of open foundation models and to get clarity on the state of the risks of releasing foundation models openly.
00:26:23
The paper is useful in the context of these broader arguments in which people are arguing cost and benefits and risks versus benefits.
00:26:32
You're focusing on the risks.
00:26:33
It's up to the people that are making those arguments one way or another to assess the benefits relative to those risks essentially.
00:26:41
Yeah, we very much did not want to offer like a guide for when you should release a foundation model because I think that decision depends very much on the specific stakeholders.
00:26:51
That having been said do you present by way of example specific foundation models and assess whether they should or shouldn't be available or provide any further guidance into the application of the framework in the context of an end assessment or do you stay away from that?
00:27:10
So I think to the extent that we focus on specific foundation models it's always tied to a specific risk or benefit.
00:27:16
So I think our main focus is clarifying what the risks are and what the benefits are and not on the foundation models themselves.
00:27:23
But like the other risk that we analyze in debt is the risk of cyber security.
00:27:28
Specifically, lots of people have claimed that cyber security risks would be hugely exacerbated by these large language models simply because we now have a tool that allows us to automatically find vulnerabilities in code.
00:27:41
Now we look back on the field of cyber security for the last 20 years and we see that automated vulnerability discovery has been much better than humans.
00:27:51
So what like someone might call superhuman for the last 20 years.
00:27:55
There have been these tools called fuzzing tools which look at specific pieces of code and try to find vulnerabilities.
00:28:02
They can do it in a way that's much faster and much broader compared to most human analysis.
00:28:07
And so in this way they can also improve automated vulnerability disclosure.
00:28:12
So why haven't we been faced with like a constant scurry of hacks.
00:28:16
But it's because defenders have access to the same tools.
00:28:19
So defenders can also use fuzzing tools to preemptively block out a lot of the harms and fix these bugs.
00:28:26
And we think the same will hold true for language models as well.
00:28:29
In fact, this is not just our hypothesis.
00:28:32
We've already seen language models like Google's palm being used for security and improving security across their own code, but as well as for open source libraries on the internet.
00:28:43
So they're using their LLM combined with a previous fuzzing tool called OSS Fuzz which used to basically automatically scan leading open source libraries on the internet.
00:28:54
And they found that using LLM can drastically improve the number of bugs caught and the number of bugs later fixed by developers.
00:29:01
So I think this offense defense balance will continue to be somewhat tilted in favor of defense even with these large language models like palm.
00:29:10
As long as we keep investing some amount of effort and energy into building tools for defense.
00:29:16
Stepping back from the paper for a moment, I'd love to have you talk about the way you think about the broader cost benefit analysis of open foundation models.
00:29:26
One of the main reasons why openness is important is because it allows safety critical research.
00:29:33
So I think a lot of reasons for our understanding of where foundation models can be used, how they can go wrong in the real world, haven't built upon a lot of work done to make these models openly available.
00:29:46
They've seen jail breaks that are applicable to open models, but transferable to closed ones.
00:29:52
We've seen new methods for prompt based attacks.
00:29:55
We've also seen new methods for defenses being developed using foundation models being raised openly.
00:30:01
So to some extent, it seems like openness is extremely valuable to fix the very problem that many critics of openness say it causes.
00:30:09
There was this very nice letter that was released by Mozilla, I think last year, which says that when it comes to AI safety, openness is not the poison.
00:30:17
It's the antidote.
00:30:19
And I think I very much believe that claim simply because the alternative is that AI and foundation models more generally are built just by a small handful of companies who are licensed to create these models.
00:30:32
Such licenses have appeared in serious policy proposals.
00:30:35
So Senators, Blumenthal and Holly have this bipartisan AI framework.
00:30:40
And I think one of the provisions of that framework is that we need licenses for AI developers to deem a few people fit for creating these models.
00:30:49
I think that's absolutely the wrong approach when it comes to AI safety.
00:30:52
For one, it is centralizing this power of deciding what's safe and unsafe in the hands of a few companies.
00:30:59
But more importantly, I think even if we ignore that part entirely, I don't think it leads to necessarily safer outcomes because it stops all of this crucial safety research.
00:31:09
And there is going to be a time when the dam would break, right?
00:31:13
There's going to be a time when compute would be cheap enough that these language models can be trained by just about anyone.
00:31:18
So like the frontier models of today are the land party models of the model.
00:31:22
Gamers at the land party would be able to train that model five years from now.
00:31:26
So restrictions based on how much compute a model needs are, in my view, wrong, because they do not leave us with the resilience of societal interventions that would be robust to the point where these models are released openly and widely.
00:31:41
Do you and your research are broadly thinking about the space look into the idea of regulatory capture by some of the large technology incumbents and that as a motivation for these anti-openness proposals?
00:31:57
The way to sort of counter back proposals is by focusing on the content of the proposals themselves.
00:32:04
So a constant theme in my work has been that I try to avoid looking at the intentions of the actors who might be sort of arguing for various proposals and instead try to focus on the content of the proposals themselves.
00:32:17
And I think in that way it leads to a stronger defense because rather than just relying on hypothesis or speculations about why someone might be doing something, I think we can actually respond to the substance of it.
00:32:28
So your argument then is openness is strong enough on its own merits and closeness is weak enough on its own merits.
00:32:35
It doesn't really matter why it's being proposed.
00:32:37
Absolutely.
00:32:38
To some extent, the interventions can be debated on their own merits too.
00:32:43
I would not go as far as to say that openness is strong enough on its own terms and closeness is not.
00:32:49
Today it's very clear that releasing language models and foundation models more generally openly seems like the right call.
00:32:55
But I think we need to keep investing in assessments of marginal risk.
00:33:00
And the other thing is like this debate is often framed as open versus closed.
00:33:04
But to be clear, no one is really arguing that there should be no closed models.
00:33:08
The only argument is about whether there should be open models or not.
00:33:12
And I think the framing this question this way means that my stance is reframe to there is a space for open foundation models in this ecosystem of closed and open.
00:33:21
So it's not open versus closed.
00:33:23
It's closed and open versus only closed.
00:33:26
How do you see this work being built upon?
00:33:29
There are many different future parts.
00:33:32
In fact, as a little backstory for how this work came about, the first draft of this paper, which now is around like 10 pages.
00:33:40
I think the final length, the first draft was 60 pages long.
00:33:43
We tried to do in-depth risk analysis of every single risk that we mentioned.
00:33:48
And at some point, we realized that we are just not qualified enough to do this.
00:33:53
We do not have the expertise needed in analyzing these specific risks to be able to confidently state what the risk assessments in that it looks like.
00:34:02
And so one of the main things I would love to see is using this risk assessment framework in different domains like biosecurity and disinformation to see what the marginal risks are.
00:34:13
And to some extent, like clearly layout this argument for why there isn't marginal risk from openness.
00:34:19
On the policy side, I think one of the main ways this framework can be helpful is in guiding what type of research policy makers find.
00:34:28
So we're already seeing AI safety institutes in the US and UK start to look at the question of openness.
00:34:33
And I think the framework can be helpful in informing them what type of research is important in urgent.
00:34:39
And I think the second area is to give policy makers who are extremely convinced that we need to lock down these models a reason to think again.
00:34:47
Because I think we really do see that there is very little marginal evidence of marginal risk from open foundation models.
00:34:55
And once we sort of hone in on this question of marginal risk, it becomes clear that at least for the time being clamping down on model weight releases might not be the best intervention.
00:35:06
To be clear, we do need interventions for biosecurity, the executive order release last year had a provision for strict screening of biosecurity materials.
00:35:14
That I think is absolutely a step in the right direction.
00:35:18
Similarly, for cybersecurity, we need resilience on the downstream surface.
00:35:23
For NCII, we need these platforms to coordinate with each other so that they can take down NCII as soon as it's posted.
00:35:29
So there is a lot of urgent action needed.
00:35:32
It's just not at the level of the AI model alone.
00:35:35
It's at the entire pipeline of how these risks materialize.
00:35:38
You were recently involved in an open letter that was published on related area.
00:35:44
Can you talk a little bit about that?
00:35:46
Absolutely.
00:35:47
So this is somewhat related to our work on openness.
00:35:50
In the open letter, we argue for a safe harbor for researchers who are investigating the risks of AI.
00:35:56
So here's what I mean by a safe harbor.
00:35:58
A lot of the legal terms and conditions that are attached to closed foundation models, as well as open foundation models, mandate that certain types of activity conducted using the model is prohibited.
00:36:10
So this might come from a place of caution.
00:36:12
Open AI might not want cyber terrorists to use their models for creating malware.
00:36:17
Similarly, it might not want people to create this information campaigns.
00:36:21
It might not want all sorts of other materials to be generated.
00:36:25
And this is all fine and good.
00:36:26
But what these terms of conditions also mean is that when researchers are investigating the risks of these models, perhaps by looking at things like chill breaking or prompt engineering, they might also run a file of these terms.
00:36:38
In imposing this one size fits all terms of condition, there is very room left for independent safety research and independent trustworthiness research when it comes to a lot of these AI models.
00:36:51
And our open letter calls for a legal safe harbor for AI research that might violate the terms of condition, but is still done in the spirit of AI trustworthiness and safety.
00:37:03
And in fact, there is a long history of protecting similar types of research.
00:37:06
So in particular, in the security community, that is, there's well established tradition of a safe harbor for security researchers who responsibly disclose these vulnerabilities to the companies in advance.
00:37:18
We argue for something very similar for AI safety and trustworthiness research.
00:37:22
If researchers are responsibly disclosing these vulnerabilities, doing good for its research, then they should be protected in terms of legal indemnification and also from their accounts being banned or removed completely.
00:37:34
In the case of the security example, you mentioned tradition, is it strictly tradition or is it legal precedent?
00:37:43
I was under the impression that it's not in frequent for security research or to get in trouble with, for example, the DCMA as part of their work.
00:37:51
Yeah, absolutely.
00:37:53
I mean, I think it is very much more than just a tradition at this point.
00:37:58
So you're right that in the last few years, security researchers have had troubles with technology companies that are investigating, but very recently, I think it was the department of justice that ruled out this notice saying that security researchers doing good for its security research are exempt from the provisions of the Digital Millennium Copyright Act.
00:38:20
And similarly, I think people have in other security areas too, fought for and won these safe harbor from company.
00:38:27
So I think we're very much relying on this long tradition of security researchers having done the work of coming up with what responsible disclosure looks like and how we should go about articulating it.
00:38:37
And we remiss if I did not mention this other safe harbor that was proposed, which is also very close to the one we proposed, which is for social media platforms.
00:38:45
So many researchers in the last few years have tried to collect data from social media platforms.
00:38:51
Let's say about their ag transparency or the transparency of how often users see certain types of posts.
00:38:57
And in a lot of cases, social media platforms have gone after them.
00:39:01
In response to such concerns, I think the night, first amendment institute wrote a letter calling for a safe harbor for independent investigations of social media, which I think was very influential in informing our line of thinking here as well.
00:39:14
And in particular, it outlined this framework for how to think about safe harbors.
00:39:19
Well, the specifics vary between these two efforts, simply because the prior effort was aimed at social media and this one is aimed at AI.
00:39:26
I think the overall spirit is very much the same that we need research access and protections when people are doing research that can that is societally beneficial, even if it is not useful to the companies or even if it might impose some liability on the companies.
00:39:41
Well, Sash, thanks so much for joining us to share a bit about your work.
00:39:49
Thank you so much for having me visit Pleasure.
00:39:55
