Um akshually, AI isn't and will never be intelligent
The qualitative difference between you and AI
AI seems smart. Yes, It’s possible to point to Google’s AI overview telling people to eat rocks (among other things) to try to prove the inverse. But for most questions, most of the time, AI seems to do a great job. It shines when asked those sorts of questions that you could figure out if you waded through the search-engine-optimized results and read a few different articles. The apparently well-reasoned results truly are a testament to the astounding technical innovation that has taken place over the last few years.
Appearances nonwithstanding, AI has issues—one, mainly.1 It isn’t intelligent. This is a striking claim, considering the name of the technology at hand, but it is also a well-founded, reasonable claim, made by people much smarter than me (who I happen to agree with). I could dive straight into the philosophical issues, like I do here, but let’s here define intelligence without terms like rational soul or substance, as those can be a bit convoluted. Once we have a working definition, we’ll look at why AI doesn’t meet that definition and what this means for AI progress as a whole.
It should be abundantly clear that you are intelligent and a calculator is not. Let’s take that as a first principle. There are objections, even to the idea that “you” exist, but this is a quite strong place to begin. Assuming this premise, we find that intelligence is a sort of capacity rather than something one does. A person (let’s call him Fred) may never learn how to do any math beyond basic arithemetic, but that doesn’t mean Fred lacks the basic quality of intelligence. On the flip side, even a calculator that can solve second order derivatives of polynomials isn’t intelligent. It’s simply following what’s been encoded into its system.
Consider what would happen if we taught Fred, who’s unfamiliar with math, how to solve a quadratic. There’s two paths we could take. We could give him a sort of formula that shows him how to get x alone (hint hint: quadratic formula). Assuming he plugs the numbers in correctly, he will end up with the right result. While this is similar to what the calculator is doing, there is one key difference. While Fred may not understand what the formula does, he understands how to follow the formula’s steps, and he chooses to follow those steps. This is, of course, not true of the calculator. To summarize, even if one takes the first path to teaching Fred how to solve the problem without teaching Fred what the formula does, some intelligence is required on the part of Fred.
The second path is to do what our teachers did in high school algebra—teach the actual mechanics behind why the formula works. The delta between Fred and the calculator is most clear here. Fred can apprehend the quiddity (the what-ness) of a quadratic equation, judge that it follows the same pattern as other quadratic equations, then reason through the steps of the formula. The calculator does no apprehending, judging, or reasoning. This is the traditional way of understanding the intellect. It is that which apprehends reality, judges it (or puts things observed into categories), then reasons from those categories.
It’s here that we come to the main issue with the definition of intelligence, where many people defining intelligence in the AI world smuggle the bathwater in with the baby, to use a mixed metaphor. We just saw that intelligence is a capacity, meaning the intellect is more foundational than the degree of intelligence which something, like an AI model, seems to have. No matter whether Fred learns anything at all or seems smart, he still has an intellect, unlike a calculator or AI model.
“But Isaiah,” the e/acc techno-optimist bro says, “ChatGPT seems so clearly intelligent, it’s passing all these benchmarks (at an exponential pace), and it’s well on its way to PhD-level intelligence. Why would you limit your definition so as to frame out something that is so clearly intelligent.” Point taken, but that’s an argument from intuition with no working counter-definition of intelligence. To the tech bro’s credit, he’s in good company. Alan Turing, the father of computer science, made the same argument for a vague definition of intelligence. He believed that the question of whether machines could think was “too meaningless to deserve discussion.” From Turing’s perspective, once a machine could pass his “imitation game,” now known as the Turing Test, one could only conclude that the machine was thinking, thereby answering the initial question without actually defining intelligence.
I like an example from Ed Feser to explain the absurdity of Turing’s view. It goes something like this: Using an imitation game to decide whether a machine is intelligent is like deciding whether a street magician can do real magic based solely on a performance seen from 25 feet away. You’d be letting the machine or magician operate convincingly in their element, then accepting that appearance as reality without getting to the mechanism behind the magic. The incoherence of the claim that AI is truly intelligent becomes clearer when you map the tech bro’s objection onto the example of the Magician: “Why would you want to investigate up close? There’s no need! Every time I come back to watch this guy, he’s even better than before! See, that’s magic.” No. This is absurd. There exists as much intelligence in an AI as magic in a magician. This even applies to AI that can pass the Turing Test.
Point being, Turing is wrong. No matter how convincing AI is at simulating intelligence, it simply is not intelligent. Let’s now answer the question as to why AI is not only is not intelligent, but cannot be intelligent in the way that we’ve defined it.
Most of the AI models you’re familiar with (Large Language Models like ChatGPT, Claude, Gemini, etc.) are predictive models. They predict what the right response to the user is based upon their training data and data from the conversation.2 This prediction can also be called “pattern-matching.” I’m not a huge fan of the autocomplete-on-steroids way of looking at the issue, but this framing is actually more accurate than calling AI intelligent. LLMs are, in a way, farther from intelligence than a calculator, becasue they don’t use formal thought processes and are mostly built on semantics. Even when you ask an LLM for the solution to a math problem, it’s not reasoning through it.
Just for fun, I asked Perplexity’s Sonar-Reasoning3 model to add five and three. The reason I asked that model specifically is that it gives the sources it uses from its training data, even for super simple questions, and it also shows its “reasoning” process. Everything between <think> and </think> wouldn’t be shown to the user normally.
From this, it’s clear that the model is operating without intelligence. It’s only been trained to answer with the most likely answer. Becuase most of the internet gives the same answer to 5+3, that’s the answer the model gives. One of its sources is an internet math forum. It doesn’t exactly inspire confidence in the ability of AI to handle more complicated tasks that actually require reasoning or deciding on the truth or falsity of something.
That’s not just me speculating. A recent study from Apple confirms that complex questions with right or wrong answers trip up AI. When AI models, even “reasoning” models, were asked to solve word problems requiring mathematical reasoning (the results of which can be extended to formal logic), the AIs failed miserably after a certain number of clauses were added to the questions. They summarize that drop in ability to solve the word problems:
This is in line with the hypothesis that models are not performing formal reasoning, as the number of required reasoning steps increases linearly, but the rate of drop seems to be faster. Moreover, considering the pattern-matching hypothesis, the increase in variance suggests that searching and pattern-matching become significantly harder for models as the difficulty increases.
The researchers also find that even when only the names of the characters in the word problems are changed, it affects the models’ success.
It is both striking and concerning that such performance variance exists when only changing proper names, as this level of variability would not be expected from a grade-school student with genuine mathematical understanding.
In plain terms, an AI that cost billions to create is worse at understanding word problems than a child. The reason this happens is that AI, again, is just predicting what the best response to the question is, without truly knowing what the important part of the question being asked is. So, it guesses that the names are as important as the numbers in the problem, and this changes its answers.
This is the achilles heel of AI, especially promised superintelligent systems. It is quite literally impossible for AI based on pattern-matching neural nets, like all the AI models you’re familiar with, to be intelligent. Yes, they can predict, but they’re farther from actual intelligence than a calculator. A working calculator will always give the right result if you input the right numbers and operations. You can be sure that, as long as you’re reasoning using a valid sylogistic form with true premises that your conclusion will also be true. The same cannot be said for AI.4
Likely, we haven’t reached the plateau of what predictive AI can do. Yet, we ought not rely on the anwers of any model like we do on human intelligence, and the more complicated, novel, and unique the questions are, the more likely that the AI model will fail to solve the problem. AI is built to provide plausibility, not truth. The human intellect is built to seek out truth through percieving, judging, and reasoning. Thus, there exists not a quantitative difference in the degree of intelligence between AI and the human intellect, but a qualitative, insurmountable divide.
If you found this interesting, be sure to follow Def(ai) on instagram.
Yes, I’m a fan of em dashes, no this isn’t written by AI.
Huge oversimplification, but that’s the essence of a Large Language Model.
None of the so-called “reasoning” AI models actually reason. They just generate more data to then make the same predictions a non-reasoning model would make. So they can be better at predicting, but they aren’t reasoning.
There’s an interesting project from Stephen Wolfram aimed at creating an AI model that does reason, but it’s not an LLM. It’s essentially just an advanced calculator, but it probably does deserve its own essay.