AI "Distillation Attacks" Are Profoundly Stupid
Apparently, the law applies to everyone else but AI companies...
Anthropic recently sent a letter to US officials accusing Alibaba of “brazenly” and “illicitly” attempting to extract its AI capabilities with the largest known distillation attacks on them to date. Anthropic flagged 16 million interactions with its Claude chatbot across 24,000 fake accounts as distillation attacks tied to Alibaba and the Chinese AI models it works with. If you have never heard of distillation, it is a relatively common AI training method where the output of a larger, more advanced model is used to train a much smaller, lighter model. When conducted “legitimately”, this is a great way of making smaller, cheaper-to-run models that are still relatively capable. But when an AI company uses the output of a rival’s more advanced model to train its small, lightweight model, it is considered an illicit distillation attack, as the original developer isn’t being paid for their work. If you have even a passing understanding of how LLM chatbot AIs work, you may already see the cosmic-sized hypocrisy at play here. But I guarantee you that Anthropic’s argument is far more stupid than you realise.
For those who don’t know why this is hypocritical, let’s start at the beginning. Chatbot LLMs need to be trained on a truly colossal amount of data. So much so that if AI labs like Anthropic or OpenAI actually paid the copyright holders of said data, it would immediately bankrupt them. So, they just take the data without paying.
Anthropic downloaded seven million books from LibGen’s “shadow library” without paying for a single one and shoved them into Claude. Reddit also sued Anthropic for illegally scraping user data from its site over 100,000 times, again to shove this data into its AI. To get around copyright laws, Anthropic bought millions of physical books and scanned them all (destroying them in the process) to create a digital copy to feed into Claude. This utilised a rather dubious loophole in copyright law, as they did technically buy the works, but the copyright holder did not agree for their works to be used in this manner and wasn’t appropriately compensated. On top of all of this, just like every other AI company out there, Anthropic has been accused of scraping virtually the entire internet for data, hoovering up a truly astronomical amount of copyrighted material, and once again, feeding it all into Claude without paying the creators a cent.
I don’t think people realise the scale of this copyright-nicking scheme. Take Common Crawl, a web scraping service used by Anthropic. Its dataset was a whopping 9.5 petabytes in early 2024 and has since added 3–5 billion new pages each month (roughly 380 terabytes of uncompressed data per month), meaning their current total filtered dataset could quite easily be well over 20 petabytes.
I want to put that number into context. Let’s say we wanted to download this dataset onto bog-standard 256-gigabyte iPhone 17. We would need 72,760 of them in order to store all 20 petabytes. An iPhone 17 is incredibly thin at just 8.8 mm, but if we stacked all of these on top of each other, it would make an iPhone tower over 640 metres tall (2,100 ft). Now, not all of this dataset is made up of copyrighted data; much of it exists in the public domain, but a considerable portion is copyrighted material, such as articles, blogs, guides, books and video transcripts.
You get the picture. If we saw this as copyright theft — which arguably we should, considering the precedent set by the horrific case against Reddit founder Aaron Swartz for ‘stealing’ scholarly articles, which led to his eventual suicide — then this would be one of the largest criminal thefts in modern history.
