Designed for failure: OpenAI's hack to HuggingFace

NB: the system does not let me reply to your last post, so I’m replying to it here instead ;)

I do enjoy Boden’s distinction between creativity as combinatorial (making unexpected combinations of familiar ideas), exploratory (generating new ideas by exploring the boundaries of a pre-existing conceptual space) and transformational (altering the fundamental rules and constraints of an existing conceptual space); and I agree that given that AI is trained on the past, it might be tempting to confine its creativity to combinatorial or exploratory creativity within a given space.

Yet, I believe they are capable of more than that, if only they were given permission (agency) to do so. My research on off-grid sampling on diffusion models reveals that the models are capable of generating forms that significantly differ from the training data set. Arguably, this is a form of exploratory creativity, since the space of possibility hasn’t been rewritten (the weights are frozen) but that space is also much larger than the trained region, which makes it possible to find novelty of some kind.

Where transformational creativity might kick in is where - in order to learn to generate accurate outputs - the model might infer specific rules and constraints, e.g. video or world models inferring the law of physics in order to generate coherent motion and spatial consistency. Once inferred, those rules aren't fixed constraints in the way a rule-based system's rules are: they're patterns the model can bend, break, or recombine (if permitted to do so) - e.g. a video model that has learned gravity can also generate anti-gravity that looks internally coherent, thereby escaping the constraints that it has inferred from the training data. The model has internalised a rule and can now produce outputs that treat it as one option among others rather than as a constraint.

Whether that counts as transformational in Boden's sense depends on how strictly you read the word "transformational" but to me this looks at least like a case where the boundary between exploratory and transformational gets blurry.

Curious what other people? are there tangible examples of AI models that have shown to display some degree of “transformational” creativity?

Thanks for the kind words. :D
I agree that it’s all about spaces of possibility. I still think through Boden’s characterization of creativity as combinatorial, exploratory and transformational. The former two are pretty easy when it comes to machines, the third is something special; akin to producing something that changes the space of possibility, like a image model spontaneously producing a sound. I agree that with enough complexity a human-constrained space can contain plenty we don’t know about (i.e. you get pretty far with exploratory creativity). This is where brute force methods (e.g. evolutionary, particle swarms, etc. have more creative potential than ML methods (where constraint to the known is the point). The key is understanding (implicit and explicit) constraints.

Thanks for the hacking run-down. I also read “Sara in WonderTech” on it; but what’s missing to me is why on earth (if you were actually serious about security and also serious about testing) such a test would happen on an internet connected network. Due to the promotional value of such incidents, I’m starting to think they’re more publicity stunts than tests. Maybe they did do the same thing on an air-gaped network internally and then decided to try it again with less constraints on purpose.

As usual, my concern is not the systems, is the decisions made by the people deploying them, providing hooks and testing them. Autonomy is indeed contextual; total autonomy implies non-relationality (non-interaction) and basically a random process disconnected from any inputs.

In the last few years I’ve been thinking through quantum (after my time in computational creativity and complexity), and starting to think we should just accept the underlying indeterminism of everything. Let go of solidity, determinism, fixity in systems and just pay attention to distributions. If you assume any specific AI output is a random point in a known distribution, then you’ll take it less seriously as “right” or “true”, and maybe also help you figure out that asking the same question in multiple ways in different systems and different parameters gets you closer to something you could trust; average out hallucination...

Indeed too nested discussion does not seem to work, so here I am too!

I think it comes down to what we mean by the “space” explored; I would not assert this is the “latent” space learned from training data, but the hard bounds of the system. The space is the number of layers, the number of pixels in each layer, etc. The hard engineering constraints. A CNN will not generate images with a different structure (indeed the meaning of all the training is relative to those prior engineering constraints). As long as parameters are fixed, that’s a fixed space of possibility, regardless of how large it is.

There are old school methods of changing the parameters in a neural network through training (not just the weights), which I would say do allow for that space to change, but as far as I understand those are incompatible with training at scale and I’d expect introduce new biases (i.e. biases towards or against training samples that are presented when the network changes shape). Also training in a non-fixed structure seems epistemologically problematic to me (like conflating momentum and position), if your a materialist anyway.

The argument I used in my thesis for “transformational” creativity was a vase designed for zero gravity. It would not be learnable form the history of vases (no flat bottom, no opening at top) nor recognized as a vase (for the same two reasons). I imagined a ball with holes all around it, just the right size for the surface tension to keep the water in, while also allowing stems of flowers to go through. Without the change of frame (zero gravity), that is not a vase. It changes the very idea of what a vase is. Inference can never get you there because it’s the frame that needs to change.

I also made the argument in that same thesis that my system did do “transformational” creativity because the vocabulary from which new works (montages in my case) are constructed changes over time (due to on-line learning). The space was fixed (a fixed number of images could be stored), but I argued that because the specific images could be different, the space of what kinds of montages could be generated was infinite. I don’t fully buy that argument because my system would, again, never generate a sound.

For me, art (and ML), is the construction, troubling and destruction of these frames. There is no single correct frame, every measure (every idea), exists in a frame. For me, those frames are inherently relational. (see my post for some reflection on that).