Zico Colter
speaker
161 appearances
1 recordings
1 series
first heard Sep 2024
last heard Sep 2024
Zico Colter’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
If this was demonstrated as a capability of a model, I would have a very hard time saying, of course, yeah, let's just release it. You know, there's no problem because we'll use it for good. That's dual use, so we'll use it for good purposes. We all know that patching software is much harder than finding exploits in software. Patching all software is much harder than finding exploits in software.
Yes, there's dual use. You can use it to secure software better, but that takes time. It's hard. I don't think we should just immediately snap to release a model that could find a vulnerability in literally any code that's out there in the world. And so I wouldn't want that to be released open-weight for anyone to use.
Now, I take some solace in the current situation we find ourselves in, which basically in the current situation we find ourselves in, at least for now, there's a constant stream of closed models that are released sometime before an equivalently capable open-weight model. And I think this is actually a very good thing.
Because my hope would be that and we've sort of found ourselves here by accident. It didn't have to be like this. I know some companies are pushing to open source more powerful models than we have ever had before right at the outset. That makes me a little nervous. But right now we're not in that world.
We're in a world where the most capable models, the first releases of them of a certain capability typically comes from closed source models. I think this is a good thing.
I think it gives us some time to essentially come to terms and understand the capabilities of these models in a more controlled environment such that we can reach a level of comfort to say, maybe not full comfort, but at least a level of comfort to say, yes, it's probably okay if we release this, a similar model open source.
And I hope, my sincere hope would be that if one of these models does really demonstrate the ability to create an exploit for any executable code or compiled code or anything else instantly, and we see that in the closed source model first, we would think a little bit about whether we really want to release this model, an equivalent model, open weight, and just for anyone to use.
I do think that the more far-fetched scenarios about sort of agentic AGI systems that start sort of intentionally acting harmful against humans, the rogue AI that decides it wants to wipe out humanity and goes about planning on how to do this.
These more, I would say, what seem to me, and I'll be honest here, far-flung sci-fi-ish scenarios here, these are often the debates we have when it comes to AI safety. I want to say two things about this. The first is that I think the vast majority of AI safety should not be about these topics.
The vast majority should be about quite practical concerns we have on making systems safer, like the kind that I've talked with you about so far. There are already massive
safety considerations and risks that are present in current systems and would certainly be present even in slightly more capable systems, irrespective and regardless of the timeframes associated with AGI and certainly the timeframes associated with rogue intelligent AI systems. However, I also don't want to dismiss this entirely.
The way I would put it is I am glad people are thinking about these problems, I'm glad people are thinking about the capabilities and even what I consider far-flung scenarios. They are good things to think about as, by the way, are much more immediate harms of AI systems like misinformation, like misuse of these things.
I think killing jobs is much more immediate of a concern than killing humans. An example I often use here to kind of try to bring a little bit of these two sides, the AI taking over the world, killing us all, and kind of the more skeptical minded academic folks, we'll say.
I see a path right now to a world in which, you know, in a few years from now, we start integrating AI models into more and more of our software. We start building it up more and more. We sort of make these things a little bit more autonomous in their actions.
We start just naturally, because software does everything for us, we start naturally kind of infusing this into all software we have, including software that handles things like critical infrastructure, stuff that controls the power grid, things like this, right?
And now all of a sudden, you have these agents that are sort of, you know, taking an active, playing an active role in doing things like controlling power grids. This leads to the possibility of, even in my view, sort of massive correlated failures that could do things like bring down power, electricity in a way that we can't restore it easily in for a large portion of the country.
And I think it's honestly not again, if we go down the wrong path, this is definitely not that impossible to imagine right now in this world where the power has been shut off, you know, we can debate and decides can debate about whether this was a bug in the system. And we should never have installed LLMs here in the first place.
Or we could debate whether this was actually the rogue AI taking over and deciding to shut off the power so it could kill all humanity. But who cares? The power is still off. This is still a catastrophic event for the country. And so we have to have a plan for how to sort of think about events like this happening. This is an example I come to often.
To a certain extent, it doesn't matter whether the AI is intentionally doing something in an evil fashion while deceiving humans, or whether this is a bug and a flaw in the system. The end effects are the same in some cases. And so we need to desperately take kind of put in structures in place that prevent these things from being possible.
Well, exactly. So yeah, we are all very familiar right now with the downsides of correlated failure, right? And imagine if that was also true of all the SCADA systems that were operating power, the power grid right now, which, you know, not impossible to believe. And the problem is that these systems, because we don't understand, really, I mean, and we don't, we don't understand them, right?
Showing 121–140 of 161 · page 7 of 9
← Previous
Next →