AI "Distillation Attacks" Are Profoundly Stupid

Apparently, the law applies to everyone else but AI companies...

Photo by Rishabh Dharmani on Unsplash

Anthropic recently sent a letter to US officials accusing Alibaba of “brazenly” and “illicitly” attempting to extract its AI capabilities with the largest known distillation attacks on them to date. Anthropic flagged 16 million interactions with its Claude chatbot across 24,000 fake accounts as distillation attacks tied to Alibaba and the Chinese AI models it works with. If you have never heard of distillation, it is a relatively common AI training method where the output of a larger, more advanced model is used to train a much smaller, lighter model. When conducted “legitimately”, this is a great way of making smaller, cheaper-to-run models that are still relatively capable. But when an AI company uses the output of a rival’s more advanced model to train its small, lightweight model, it is considered an illicit distillation attack, as the original developer isn’t being paid for their work. If you have even a passing understanding of how LLM chatbot AIs work, you may already see the cosmic-sized hypocrisy at play here. But I guarantee you that Anthropic’s argument is far more stupid than you realise.

For those who don’t know why this is hypocritical, let’s start at the beginning. Chatbot LLMs need to be trained on a truly colossal amount of data. So much so that if AI labs like Anthropic or OpenAI actually paid the copyright holders of said data, it would immediately bankrupt them. So, they just take the data without paying.

Anthropic downloaded seven million books from LibGen’s “shadow library” without paying for a single one and shoved them into Claude. Reddit also sued Anthropic for illegally scraping user data from its site over 100,000 times, again to shove this data into its AI. To get around copyright laws, Anthropic bought millions of physical books and scanned them all (destroying them in the process) to create a digital copy to feed into Claude. This utilised a rather dubious loophole in copyright law, as they did technically buy the works, but the copyright holder did not agree for their works to be used in this manner and wasn’t appropriately compensated. On top of all of this, just like every other AI company out there, Anthropic has been accused of scraping virtually the entire internet for data, hoovering up a truly astronomical amount of copyrighted material, and once again, feeding it all into Claude without paying the creators a cent.

I don’t think people realise the scale of this copyright-nicking scheme. Take Common Crawl, a web scraping service used by Anthropic. Its dataset was a whopping 9.5 petabytes in early 2024 and has since added 3–5 billion new pages each month (roughly 380 terabytes of uncompressed data per month), meaning their current total filtered dataset could quite easily be well over 20 petabytes.

I want to put that number into context. Let’s say we wanted to download this dataset onto bog-standard 256-gigabyte iPhone 17. We would need 72,760 of them in order to store all 20 petabytes. An iPhone 17 is incredibly thin at just 8.8 mm, but if we stacked all of these on top of each other, it would make an iPhone tower over 640 metres tall (2,100 ft). Now, not all of this dataset is made up of copyrighted data; much of it exists in the public domain, but a considerable portion is copyrighted material, such as articles, blogs, guides, books and video transcripts.

You get the picture. If we saw this as copyright theft — which arguably we should, considering the precedent set by the horrific case against Reddit founder Aaron Swartz for ‘stealing’ scholarly articles, which led to his eventual suicide — then this would be one of the largest criminal thefts in modern history.

So, how are AI labs getting away with this? Well, they argue that this process falls under ‘fair use’ as they are being ‘transformative’ with the copyrighted material.

In copyright law, people are allowed to use your copyrighted material as long as it is ‘transformative’, which is classified as ‘fair use’. This is why YouTube movie critics are allowed to use extensive clips from the films they discuss. They aren’t reproducing the material, and their use of it isn’t directly competing with the original work (which is critical). Instead, they have ‘transformed’ it into something different by commenting on it.

Now, here is the problem with the AI labs’ argument that using copyrighted data to train an AI should be classified as ‘transformative’. LLMs can fully reproduce their training data verbatim. Moreover, LLMs can only replicate the patterns present in their training data. So, if you want a chatbot to be able to produce recipes, you train it on a ton of recipes. This means that AI output trained on copyrighted data directly competes with the original copyrighted material. It will negatively impact the revenue and value of the original work.

If the laws of the land were actually used as intended to protect people, this ‘fair use’ argument would have been laughed out of court. As it stands, those in power wield and bend the law to protect capital, not people, which has meant they have let off AI labs scot-free.

But doesn’t this argument set a precedent that completely undermines Anthropic’s stance against distillation attacks?

If training an AI on data counts as transformative enough to be fair use, then doesn’t Alibaba’s distillation of Anthropic’s model fall under the same umbrella?

If the law were applied consistently and the output of a near-trillion-dollar AI had the same legal protections as living, breathing people’s work, then by Anthropic’s own argument, no wrongdoing has occurred here.

There is a second issue, though, as Google recently discovered. You see, if training AI on copyrighted data counts as transformative and the machine isn’t just plagiarising and regurgitating what it was trained on, then the output is considered new material, and the AI lab can be held responsible for its output. This is the same way that our hypothetical YouTube movie critic can still face a slander lawsuit. Google found this out recently in court, where it was ruled that it can be held responsible for any damaging inaccuracies found in its AI search summaries (read more here).

Likewise, Anthropic’s own argument against Alibaba means that it is also arguing for itself to be held responsible for any damaging errors its AI produces. Now, I don’t know if you have used an LLM recently, but these things get stuff horribly wrong all the time! I can assure you that Anthropic does not want to be held responsible for Claude’s mistakes, because that would be devastating.

Okay, so let’s say that Anthropic acknowledges these issues, U-turns, and drops this letter-writing campaign against Alibaba. That would heavily imply that AI training is not transformative and doesn’t fall under fair use. Again, if the law were enacted ethically, consistently and properly (i.e., for the benefit of the people, not capital), then that would mean that Anthropic and all other AI labs would have to pay billions of dollars in compensation to copyright holders they have stolen from.

In other words, this is a catch-22.

Either AI training is fair use, which means distillation attacks are perfectly legal, even though it will pop the AI bubble. Anthropic, OpenAI and other AI labs are only valuable because their AIs are seen as unique and owned by them, meaning customers have to go to them to benefit from their AI. But if distillation attacks are legal, that is no longer the case, as rival AIs can provide an equally good model that costs less to use. The exclusivity and value of AI labs’ proprietary tech dissolves to functionally zero, as it no longer becomes theirs exclusively, and the justification for their near-trillion-dollar valuations evaporates.

Or distillation attacks are illegal, which means AI training isn’t fair use. That would mean any copyright holder who has shared their work online, or shared anything online, which is essentially all of them (including you; if you have posted something online, you technically have copyright over that ‘work’), deserves compensation for their work being used to train these giant LLMs from the likes of Anthropic and OpenAI. Such compensation could balloon into the hundreds of billions of dollars, particularly when many copyright holders have been materially harmed by AI encroaching on their work (such as musicians).

For the AI industry to exist as it currently does, it requires an insane legal contortion. It requires systematic hypocrisy to deflect accountability on all sides to retain its precarious position. Now, in our current insane political landscape, this hypocrisy is evidently possible. This is a lawless neoliberal hellscape where what is left of the law of the land is bent to protect capital and leverage and brutalise the people who stand in its way. But, if a modicum of sanity prevails, if this chaos is toned down, if this epidemic of unreality is replaced with grounded reality, and if the law is actually applied consistently for the protection of the people — the very bedrock of democracy — then the AI industry crumbles. The AI industry is inherently betting that today’s reign of idiocracy and oligarchic power won’t end. I hope with all my heart you haven’t lost hope and, like me, believe this era of brutal stupidity will soon come to its natural conclusion. If you believe that is the case, then where do you think the AI industry is going?

---

Thanks for reading! Everything expressed in this article is my opinion, and should not be taken as financial advice or accusations. Don’t forget to check out my YouTube channel for more from me, or Subscribe. Oh, and don’t forget to hit the share button below to get the word out!