The Death Of Consumer AI Is Coming

They can't have their cake and eat it too…

Photo by Jr Korpa on Unsplash

AI is in a really weird place right now. The public’s collective sentiment towards the technology is at an all-time lowand is somehow still going down. Corporations have only just discovered how expensive and useless AI is and are poised to majorly U-turn on their AI rollout plans (read more here). Yet valuations of AI companies are through the roof. For example, OpenAI was only worth $29 billion back in 2023. Since then, its annual revenue has increased by roughly seven times, but it is expected to be valued at over $1 trillion in its coming IPO, or 34 times its 2023 valuation. There is a tension here that is begging to break, and it might soon. You see, over the past few months, people have put actions in motion that could lead to the death of consumer AI.

Okay, that is a bold statement, so I should probably define what I mean by death. I don’t mean the technology will no longer be available or used. I mean that the current vision of consumer AI — where it sits between us and what we want at every conceivable place possible — dies. LLMs, chatbots, and people dabbling with AI will most certainly still exist; that isn’t up for debate. But the consumer AI business model that this entire bubble is partially predicated on will no longer be viable.

If you have been paying attention to the news, you might already know where this is going.

Google recently lost a court case regarding its AI-generated search summaries in Germany. The court ruled that these AI summaries are the company’s own content and are therefore legally liable for any false claims the bot spits out. The implications for Google are immense, which we discuss soon, but this will also seriously impact other AI companies. For example, xAI’s proclivity for producing unconsensual sexualised images or even CSAM could be framed as xAI’s legal liability and not the users’.

Now, Google plans to appeal this ruling, but I don’t rate their chances, as any defence they could make is nonsensical from a copyright point of view. AI labs have been able to get away with stealing vast reams of copyrighted data to train their AI under the guise of ‘fair use’. To qualify the unpaid use of copyrighted material as ‘fair use’, you need to transform the material and make it something different. This is why YouTube film critics can use clips of films without facing a copyright claim, because they are transforming the material into something fresh, rather than just reproducing it. AI labs like Google, OpenAI and Anthropic have claimed that training an AI on copyrighted material counts as transformation. But that, in turn, means that they are the originator of the AI’s output and are therefore liable for whatever it claims. Again, if a film critic YouTuber were slanderous, they would personally be liable for it, rather than the creator of the movie they clipped. So, if Google wants to claim that the AI isn’t transformative and that it simply regurgitates what it finds in its training data, then it will undermine the entire industry’s argument of fair use. And boy, if you think the AI industry’s financial woes are bad now, just wait until they have to pay the people they stole from!

But I can see why Google has appealed, because they have to if they want to continue rolling out AI in the same manner.

Ultimately, AIs ‘hallucinate’. I hate that jargon — it’s just humanity anthropomorphising a soulless machine, which clouds our judgement. AI ‘hallucinations’ are just errors. AI is a statistical model, and that means, statistically, it will get things wrong. New benchmarks have shown that even the best AI models still ‘hallucinate’ 22% of the time, and those models are extremely expensive to run. The lighter, cheaper models used for things like Google’s AI search summaries have a considerably higher hallucination rate.

You might think that ‘hallucinations’ can be solved by pouring more training data and compute power into these AI models. Indeed, a decade ago, there was talk of AI potentially having ‘emergent properties’ once it scaled that would totally eradicate issues like hallucinations. Sadly, that was just science fiction. OpenAI researchers recently discovered that, no matter how much data or power you put into these machines, the hallucination rate won’t be meaningfully reduced from today’s levels. In fact, they found no viable way to reduce hallucinations or even mitigate them.

And it isn’t just OpenAI that has discovered that hallucinations are here to stay. Vishal Sikka and his son Varin Sikka published a paper which claims to mathematically prove that AIs “are incapable of carrying out computational and agentic tasks beyond a certain complexity”. In other words, the maths says AIs will always hallucinate. Likewise, there is a considerable amount of research into the efficient compute frontier, which describes how AIs need exponentially more data and compute power to increase their accuracy at a linear rate. This basically makes a hallucination-free AI impossible, as there simply isn’t enough data or computing power available. Then there is the Floridi Conjecture, which posits that AI systems can either have great scope but no certainty or a constrained scope and great certainty. LLMs like ChatGPT and Google’s search summaries are incredibly broad applications for an AI, meaning they can’t have great certainty, which is just another way of saying they will constantly hallucinate.

In layman’s terms, all the science points to the fact that Google can’t prevent its search summary from hallucinating.

This makes the German ruling a bit of a problem.

How can Google stop their AI search summaries from spraying out misinformation and rendering them liable for whatever damage is caused?

I mean, sure, they could attempt the arduous task of ensuring the AI is exclusively trained on trustworthy sources and only summarises them, rather than pulling from places like Reddit (which is how we got the famous incident of this AI stating you should put glue in your pizza and eat rocks). But even then, these AI search summaries will hallucinate at a staggering rate and, thanks to this ruling, leave Google liable for their mistakes.

This is why this ruling is potentially catastrophic for consumer AI.

The current business model for consumer AI is to use it as a filter through which we access information. The Google AI search summary is a great example of this. You don’t actually click and read the website with the answers — the answer is filtered through the AI. In fact, most consumers use chatbots like ChatGPT effectively as search-engine-style summary bots, like Google’s, because these bots are useless at agentic tasks. This approach enables Big Tech to implement AI almost everywhere and force people to use it, driving up metrics and, in theory, driving up revenue.

However, this system only works if those providing the AI aren’t held accountable for its errors/hallucinations. As soon as even a modicum of liability is imposed, this entire model becomes legally unviable.

Because consumers hate AI more than they use it, and this ruling potentially sets a damning precedent for Big Tech (that could easily impact them globally), the trap is set for the death of consumer AI. Consumers do not want AI standing between them and what they want, and using AI in this manner could soon place enormously expensive legal liabilities on AI companies, who already have colossal legal and financial woes. If legal systems hold their ground and actually hold these companies liable for the mistakes their tools make, then there is a reasonable chance that consumer AI in its current guise will disappear. However, in today’s deranged world, that is sadly a very big “if”.

---

Thanks for reading! Everything expressed in this article is my opinion, and should not be taken as financial advice or accusations. Don’t forget to check out my YouTube channel for more from me, or Subscribe. Oh, and don’t forget to hit the share button below to get the word out!