The Trouble With AI

Neither energy consumption, nor human unemployment or paper clip maximization

Imagine an alternative reality where everyone was happily living in a pre-ChatGPT world except for one guy. This guy secretly had access to a powerful state-of-the-art LLM. No one else would even suspect that such technology exists.

Would people like reading his essays and books?

They probably would. He would most likely win literary prizes and become a bestselling author given his sheer output would make James Patterson and Danielle Steel look lazy.

But why then, is everyone increasingly allergic to anything that even remotely smells like it was created using AI?

As the little thought experiment above illustrates, the issue certainly isn’t that AI is producing bad writing per se.

I still vividly remember when I first prompted a fourth-generation model to rewrite something I had written and thought “This is really good. This is much better than anything I ever could have written.”

I felt excitement and a tiny bit of existential angst.

If you never encountered something written by AI before it’s pure magic. It doesn’t seem like AI-slop.

But now when I look at the same AI output I can only think “Ugh”. The flaws and patterns are so, so obvious now. It doesn’t matter what model I use or what prompt I try.

Everyone loves creating with AI. No one loves consuming what AI creates.

But why? Why don’t we get excited when we read something written by this superhuman alien intelligence? Why do we feel disgust?

AI is full of shit

If you use AI tools regularly, you quickly figure out that they have zero shame telling you bullshit.

Hallucinations always creep in. You have to keep your guard up.

Most frustrating, there is no sign this is getting any better. Also getting angry at an LLM or explaining that this is bad behavior has zero effect. All you ever get is a cheery “You’re absolutely right!”.¹

AI hallucinations are a huge issue because, famously, the “amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce”.

This was an estimate from the pre-LLM era. Now with AI the cost of producing bullshit has gone to virtually zero while the effort required to refute it has remained mostly constant. So the ratio is more like 100x or 1000x now.

AI is full of fluff

Besides straight up lies, LLMs can’t stop producing vacuous phrases like “It’s not gradient, it’s texture”

The human mind has a strong tendency to maintain a sense of coherence.

When we look at an impossible object, we do not realize immediately that they don’t actually make sense. It takes effort to spot the impossibility of these objects.

source

Analogously, when you’re reading AI-slop full of vacuous phrases you do not immediately notice this. They do read plausible-enough for your mind to not stumble upon them on a first reading. But that doesn’t change the fact that they do not make any sense.

The situation is effectively like the one Isaac Asimov describes in Foundation when Mayor Hardin reveals that he secretly recorded the imperial envoy Lord Dorwin and ran his days of reassurances through logical analysis:

“That,” replied Hardin, “is the interesting thing. The analysis was the most difficult of the three by all odds. When Holk, after two days of steady work, succeeded in eliminating meaningless statements, vague gibberish, useless qualifications—in short all the goo and dribble—he found he had nothing left. Everything canceled out. Lord Dorwin, gentlemen, in five days of discussion didn’t say one damned thing, and said it so that you never noticed.

Just like with other types of bullshit doing the hard work of staring at the phrases long enough for them to evaporate into thin air is a ton of effort. So a much saner approach is to simply stop consuming anything that contains them.

AI has no grip on reality

Reading is rarely just about information transfer. Everyone knows that books don’t work. And yet, we’re clearly getting something out of reading them.

Who wrote something and how they wrote it is just as important as the set of facts they are writing about.

Relevance realization is the dynamically adaptive process of making and breaking frames to find an optimal grip. But finding that optimal grip typically requires more than a mere statement of the facts.

Knowledge is four-dimensional. Besides propositional knowledge, we need perspectival, procedural, and participatory knowledge to fully grasp something.

We need stories and analogies that provide alternative frames until it all “clicks” and starts to make sense to us.

The trouble with current AI models is that they have no grip on the world. They have no experiences of their own. It’s all Fugazi.

State-of-the-art LLMs might do a reasonably good job at parroting propositional knowledge. But whenever they are outputting more than mere facts, it feels off since they lack the other types of knowledge completely. This is why the analogies they come up with often make no sense.²

AI makes you dumb

Your inputs determine the quality of your thoughts and outputs. If you consume too much AI slop your mental landscape will start to consist primarily of vacuous junk. Because, let’s be real, no one has the time or energy to check all sources and think hard if every single analogy makes sense. So your thinking becomes fuzzy and delusional.

But that’s just consumption. Creating with AI is just as bad.

The example below is the perfect visualization of what happens when you use AI to “express your idea”:

source

LLMs take your inputs and return a polished, sanitized version that seems plausible enough. And if you’re not careful, you do start to believe that this is a perfect expression of the idea you had in your mind all along. Except that it isn’t. It’s vacuous slop.

When you’re writing, the process, the struggle for the right words, is the whole point.

The fantasy that you have an idea and then just write it down is just wrong. Deep thinking happens as you’re writing.

I virtually always discover that the idea I started writing about actually makes no sense. But I usually do discover three new ideas along the way. These ideas often don’t make much sense either but when I repeat this process enough times I eventually encounter one or two ideas that do not immediately collapse.

Another benefit of typing yourself is that you are forced to read and re-read what you’ve written. You have to build up and keep a complete model of what you’ve written in your mind to make sure it all makes sense. A first draft of human written text is always unpolished. So you have to keep re-reading and editing it until you have something you are not ashamed of sharing. This process of editing is often the source of my biggest insights. By repeatedly examining the ideas I’ve written about and looking at them through alternative angles to find the best fitting one typically reveals connections I had previously missed.³

So in short, taking a “shortcut” via LLMs means you’re missing the point of it all.⁴

And it also means that anything written with the help of AI is usually not worth your time no matter how hard the author tries to convince you that it’s all their ideas and AI just helped to express them nicely.

AI sucks the joy out of everything

I used to love coding. Translating ideas into lines of code is immensely satisfying. There are few better feelings than waking up in the morning with the exact solution to a coding problem you wrestled with for days magically appearing in your mind.

But this has all changed when I started using LLM-based coding assistants like Cursor and Claude Code. I was never a great coder. So I do feel like these tools make me more productive. But the joy is largely gone.

I used to spend days coding in flow state. Flow state is essentially an insight cascade. You solve problems of all shapes and sizes, your fingers float over the keyboard, time flies, and at end of a session you have no way to explain how you created this. You feel like something took over you. But deep down you always know this is your creation.⁵

When creating with AI the subjective experience is pretty much the exact opposite. There is no flow state, no insight cascade since you no longer hold a complete enough model of the codebase in your mind. You feel like you repeatedly pull the lever of a slot machine, hoping the LLM will get it right this time, only to be disappointed again and again and gradually lowering your standards. Instead of proud you feel dirty because deep down you know you didn’t create any of it and it’s all pretty crap.

No matter how hard you try to hide it, this nagging feeling will always shine through. People can sense if you were truly excited when you created something or whether you merely went through the motions for the sake of producing an output.⁶

---

In summary, we feel disgust whenever we smell LLM outputs because it’s a useful heuristic. Thanks to the repeated exposure to LLM outputs our mental immune system has been trained to detect it and reacts accordingly.⁷

Whenever you sense the cheery demon they are summoning in San Francisco had its hands in creating something, your alarm bells starts going off. You now have to carefully check every single claim, every analogy to avoid filling your mind with junk and you immediately know reading further most likely won’t spark any joy. You just know there is no there there.

None of this means that LLMs are useless. They can be wonderful tools, for example, as advanced search engines, translation tools, or grammar checkers.

But they do more harm than good when used otherwise and we overestimate their usefulness.⁸

1. Imagine you had a friend acting like this. Great guy. Fun to talk to. But 10% of the stuff he told you was a complete lie.

You, of course, would explain to him that this isn’t cool. Every time he would tell you “Yes, I agree. This will not happen again.”. And every time he would keep lying at the same 10% rate.

Would you remain friends with him?

2. Even though perspectival, procedural, and participatory knowledge can’t be transferred directly through words, they are what’s enabling all the content besides the facts.

So reading an LLMs explanation is analogous to reading a book by a charlatan who isn’t actually practicing what he’s preaching and simply made up all the stories he’s trying to pass off as his own experiences. You can just sense that something is off.

3. This is also true when you use AI to code. Since you only have a low resolution map of the codebase in your mind you have significantly fewer insights.

This also explains why the indie hackers (bootstrap online entrepreneurs) community is dying even though naively you would assume it would explode thanks to AI. In theory, AI should unleash a huge amount of latent creativity. But pretty much we’re seeing the opposite because vibe coding is making people less creative.

4. This is also why I’m sceptical of all the hype around AI advancing mathematics. AI might help to prove or disprove certain theorems. But all the real progress comes from the tools people invent when they’re trying to tackle certain theorems and the understanding they develop along the way. As Bill Thurston put it, “The product of mathematics is clarity and understanding. Not theorems, by themselves.”

5. The Greeks didn’t say someone was a genius but that someone had a genius. A genius is a spirit that visited you. The experience was of something flowing through you. They called this state eudaimonia which translates to “being well-possessed” and considered it the highest form of happiness.

6. A good analogy is the difference between eating at a restaurant that cooks from scratch and eating at one that serves reheated Sysco slop.

This also ties in with research on the effort heuristic. Roughly, a big factor in assessing the quality of an object is “determined from the perceived amount of effort that went into producing that object”. LLM smell indicates low effort and hence low quality.

7. We have to be grateful that LLMs are so widely and easily accessible. If only a small group of people had access and they wouldn’t disclose their LLM usage, our mental immune system couldn’t have adapted so quickly.

8. An underdiscussed aspect of LLMs is just how addicting they are, probably on the same level as nicotine. Whenever I’m stuck for more than a few seconds, I hear a little whisper "Why not use AI? Let’s see how Claude would continue here.”