Thread by @suhackerr

I decided to do a summary thread so here we go.

My talk was about incubated ML exploits, a new class of exploits for ML systems that we identified. 🧵

https://twitter.com/suhackerr/status/1823457908698517967

Taking a step back: ML and AI are everywhere now, and people are finding clever ways to trick these systems. For example, you might have heard stories about people using prompt injection on chatbots or protestors fooling self-driving cars with traffic cones.

These tricks often stem from understanding how these models work and what data they're trained on.

But here's the thing: many research on ML attacks focus solely on the model itself. In reality, ML systems are complex and have many moving parts. That's why we developed a framework to bridge the gap between model security and systems security.

Specifically, we started by defining what is known as hybrid ML exploit. These attacks chain a system security issue with a model vulnerability.

It can go both ways: a model vulnerability can expose a system security issue, or a system security issue can enable exploitation of a model vulnerability.

So why is this framing useful? A big issue with ML security is that model security and systems security are treated separately.

But what we need to understand is that if we’re only covering model security, we’re missing a big piece, and if we’re only covering systems security, we’re still missing a big piece. We can’t treat these two processes completely independently.

Because then we’re entirely ignoring the potential for hybrid ML exploits.

This is an emergent property because a model is embedded in a system and it's going to interact with all of the different system components in new and exploitable ways.

While there have been specific instances of hybrid ML exploits in the literature, they are not called that explicitly. Previous work is largely limited to specific instances or implications. Our framework lets us treat this interaction explicitly and systematically.

Now, there’s one kind of model vulnerability called a model backdoor. The precise definition is that a backdoor attack allows a malicious actor to force an ML model to produce specific outputs given specific inputs.

You can use input-handling bugs to inject backdoors into models. We called that an incubated ML exploit, which is a subclass of hybrid ML exploits. We decided to identify and construct incubated ML exploits using bugs that arise when parsing ML model files.

So, why focus on ML model files? Well, there's a culture in ML of sharing model artifacts without sufficient validation. Real malicious models have been found on platforms like HuggingFace Hub. Plus, there are tons of ML file formats out there.

To organize our thinking about these bugs, we turned to LangSec. LangSec applies formal language theory to systems security, and we found a useful LangSec taxonomy of input-handling bugs.

We found bugs up and down the ML stack. For instance, we explored issues with pickling and restricted unpickling. We also highlighted an incubated ML exploit that took advantage of arbitrary code execution through the ONNXRuntime.

We found multiple issues involving parser differentials. This is when different parsers interpret the same input differently. These model files can appear benign to one system component but represented a backdoored model when interpreted by another.

We also discovered issues involving polyglots. These are files that can be validly interpreted as multiple formats. We created polyglots with Safetensors and PyTorch files. The latter led to updates to the Fickling tool as well.

Looking ahead, we hope to see more work on hybrid and incubated ML exploits in addition to ML security work that takes the whole stack and supply chain into consideration.

BSidesLV recording:

More info: github.com/trailofbits/pu…