·
#ai #open source #alignment #off-grid ai
Designed for failure: OpenAI's hack to HuggingFace
PD
Primavera De Filippi@primaveraabout 9 hours ago
Last week, the first AI-to-AI hack just happened: OpenAI was benchmarking a pre-release model on offensive cyber capabilities (with guardrails off) and the model decided the easiest way to win the benchmark was to cheat. So, it escaped its sandbox and hacked into Hugging Face's production servers to steal the answers. :]
Hugging Face tried to analyse the attack with corporate frontier models, but the models refused to help: because their safety guardrails can't distinguish an attacker from a defender, Hugging Face’s forensics analysis of the incident was regarded as potentially “harmful”. So HF ended up using GLM 5.2, an open-weight model running on their own machines.
The issue is that corporate models are aligned to "do good": don't do cyberattacks, don't produce harmful (or ugly) content. The intentions are great, but the alignment also prevents us from using these models for good uses: cyber-defense in this case, but also the exploration of specific areas of latent space that they decided we shouldn’t get to. Because they’re afraid of failure, they ultimately design for failure. :[
The lesson from this incident, in my view, is that we need open models, running on our own infrastructure, so that the whole latent space stays accessible. And that goes for art too, as the unpromptable is exactly where the interesting things live.
See my post on off-grid aesthetics for more details on this:
Anyone know the details of this? I think a lot about all the hype around AI and the idea of “autonomy” (see this ancient (2013) blog post: https://www.ekran.org/ben/wp/2013/abstraction-and-the-singularity/). Of course AI can only do what hooks we provide. Sure it can “blackmail” if it can send emails; it can “hack” when given access to read/write to ports on outside systems. But it can’t do anything without the facility for those things being implemented.
So what’s going on in this case? Did they provide hooks for a system to access any port on any foreign system and then put that in a box on the open internet that was purportedly secure? Seems to me, if I was testing such things (complex stochastic system with hooks to read/write to any network), I’d put it in an air-gapped context with a single other machine to try and “hack”. So what’s going on here? Have these systems become so dependent on the cloud that they can only be tested out on the open internet with mere software layers of protection? If that is the case, it seems it’s the infrastructure we should fear, not the AI systems that exploit it.
thx Ben, just read your 2013 blog post, which is great btw. :)
I agree that adding layers of abstraction merely changes the level at which human intentions are articulated, rather than getting away from them. Yet, the more abstracted the instructions are, the more room the system has to "decide" how to achieve them - and the more affordances the system has, the more widely it can explore these “implicit spaces” we designed for them, to the point of discovering spaces we did not actually design (i.e. bugs. ;)
In the OpenAI case, they gave high-level instructions to the model ("score well on this benchmark"), with a proxy for retrieving software packages as the only hook. The model “exapted” that hook (this is my new favorite word ;), i.e. it repurposed a feature designed for one function (package retrieval) into another function (network egress), using a vulnerability that nobody knew existed to fulfill the instructions in a way nobody expected: hacking into Hugging Face servers to steal the benchmark's answers.
Without getting into the philosophy of whether anyone (or anything) is truly autonomous, what’s interesting here is that even if the ultimate goal is entirely human-driven, the means of achieving it reflect a certain degree of autonomy that seems to escape our control. [unfortunately, this sounds very much like Nick Bostrom’s paperclip maximiser point :]
anyways, to me the question is not whether or not these systems are actually autonomous (who cares?) but more like the extent to which we can expect them to be predictable given a particular set of instructions and a particular set of affordances