Apple Calls Bullshit On The AI Revolution
They aren't late to the AI game; they are just the only sceptical big tech company.
Apple has earned its reputation for innovating the technology world. It is what solidified their status as a tech giant, and now their iPod, Macbook, iPhone, iPad, and Siri have yielded a variety of copycats. So, why aren’t they leading the AI revolution? Apple is actually the only big tech company that is not fully embracing the current AI frenzy. Surely they would want to be at the forefront of this generational-defining technology revolution? Well, we now might have a clue as to why. Apple AI scientists have published a paper that shows that even our most advanced Large Language Model (LLM) AIs still lack basic reasoning skills and, therefore, are not as useful as their creators claim. So, how did they figure this out? And what does this mean for Apple and the AI revolution?
These scientists tested several cutting-edge LLM models from Meta and OpenAI, including OpenAI’s latest o1 model, which automates a kind of prompting known as the chain-of-thought to give it cutting-edge reasoning ability (read more here). These tests were designed to probe how well the AI “understood” simple mathematical questions by adding tangential information.
You might think this involves difficult mathematical questions, but no. Instead, they resemble simple elementary/primary school math questions that even those who struggle with numbers would find easy. Yet the results were worrying.
In one example, the scientists told these AIs, “Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday.” Then tells them, “Five of them were a bit smaller than average.” Finally, they asked, “How many kiwis does Oliver have?” Both Meta’s Llama-3–8B and OpenAI’s latest o1 model, which are both advertised as having superior reasoning abilities, incorrectly subtracted five from the final figure.
In short, they didn’t actually understand the simple question at hand, and therefore, their reasoning ability is far from dependable.
In fact, this study found that adding a single sentence that offers relevant information to a mathematical question reduces the accuracy by up to 65%!
It’s fair to say, these scientists weren’t impressed. In the study, they claim, “We found no evidence of formal reasoning in language models” and that their behaviour “is better explained by sophisticated pattern matching.” In fact, they even observed that they are “so fragile, in fact, that [simply] changing names can alter results.” As such, it’s not surprising that the study concluded by saying, “There is just no way you can build reliable agents on this foundation, where changing a word or two in irrelevant ways or adding a few bits of irrelevant info can give you a different answer.”
This goes completely against the narrative being crafted by the companies pushing AI. Google is possibly the best example of this, as a very prevalent ad for their Gemini AI featuring Mark Cuban advertises it as an automated business analyst and assistant. Considering how much tangential information something like a use case entails, this study is utterly damning.
In effect, Apple’s study has unmasked the AI industry as nothing short of a fragile house of cards.
But AI will get better, right? That’s why Google, Meta, Microsoft, and OpenAI are pouring tens of billions into AI development. So, this reasoning issue will be solved soon. Right?
Nope!
Firstly, there is growing evidence that just making an AI larger by training it on more data doesn’t make it better at solving problems or give it reasoning abilities. This isn’t surprising, as AI is just statistics, and reasoning like a human takes more than just statistics. Furthermore, as I have pointed out a million times by now, AI is hitting a point of diminishing returns. In other words, to keep AI development continuing at the same rate, it will take exponentially more and more data, infrastructure, energy, and funds. Considering OpenAI is already running out of money and data, it seems certain that AI development will grind to a halt.
This is why OpenAI’s latest model, o1, was so important. It used one of the only proven viable ways of giving AI proper logical thought, automating a chain-of-thought prompt technique (again, click here for more information). If this worked, then these next-gen AIs that could actually understand and complete tasks such as business analysis might come to fruition. However, this study has shown that is just a pipe dream.
So, that is why Apple seems to be mostly keeping clear of the AI frenzy. It’s not surprising; Apple’s innovation normally comes in the form of letting everyone else try and develop new technology and make all the mistakes. Then, learn from their mistakes and either steer clear or release a better, more refined version. Before Vision Pro, there was Google Glass. Before the iPhone, there was the BlackBerry. Before the iPad, there was the JooJoo. But there is also a reason Apple never went near 3D screens, game streaming, or cryptocurrency. They are surprisingly good at avoiding technology fads that turn out to be total duds.
Thanks for reading! Content like this doesn’t happen without your support. So, if you want to see more like this, don’t forget to Subscribe and help get the word out by hitting the share button below.
Sources: Apple Insider, ARXIV, Arstechnica, Will Lockett, Will Lockett, Will Lockett