I’ve been learning a lot about AI over the last few months. My company's focus on developing expertise has encouraged me to try new things, and to see where I could apply it. I realize we’re dealing with truly transformational technology, and my own inclination is to try to understand it the best I can. That means testing it, interrogating it and trying to break it. I would be remiss not to mention up-front that these Large Language Models (LLMs) were trained on, among other things, the life’s work of artists and content makers whose work was used without their consent and against their express wishes. There are also very real and very substantial energy and water concerns that arise from AI and need to be dealt with at a public policy level. And there are companies where people are losing their careers because companies are shifting work from Humans to AI to save money. I think there are genuine solutions available to some of these problems, but not all of them. I believe that many of these problems originate from a misunderstanding of what the technology is, and others come from the speed of AI outstripping public policy. The idea that AI is like an intern gives it far too much credit. It’s not an expert either, though it is a very useful tool, properly applied. What I’ve done so far I started out with chat prompts, using it as a kind of search engine. I quickly realized I could use copilot to create tables comparing options for a product (e.g. comparing the price, dimensions, name and country of origin of a new chair I want to buy). Then I tested how it could generate images. I was hoping to get a visual representation of how it interpreted phrases. That was eye-opening because if I prompted “a Renoir-sketch style hockey player surrounded by 42 cans of diet soda” I would get the hockey player, Renoir style, and 4 cans of soda. 4 cans instead of 42 sounds like a petty complaint, but it is an example of failing the instruction, not merely doing it badly.
Since then I’ve used copilot studio to build my own “Agents” and through that process come to understand what an “Agent” is. I’ve now built my own home AI interface, and through that process come to understand what the parts of an AI are. The more I learn, the more I’m convinced of one thing: Human language is the biggest problem facing this field. Artificial Intelligence was a great metaphor to get this technology off the ground, but it now does more harm than good. I suppose it’s probably borrowed from sci-fi, but sci-fi just had to posit it. Authors didn’t have to build it, and they didn’t have to understand it. “AI” implies understanding, comprehension, identity. An AI chatbot is a probability engine (LLM) with a keyboard attached. It uses your input for context and then choses the most likely next word based on what it has seen before. And then the next word and so on until it’s done. It does not know 2+2=4, but it has input of people referencing 2+2 often enough that it’s almost certain to answer 4. It doesn’t calculate or reason things out, it predicts what usually comes next. AI is very slightly worse than a calculator at answering a simple arithmetic question because it doesn’t perform arithmetic at all. Some would argue it doesn’t matter as long as it gives you the right answer. I disagree in part because… “Hallucination” is misleading. Hallucination is a word chosen to cleanly communicate an AI chat giving an incorrect response. It won’t try to “respond to” or “answer” your question. It is using learned patterns to build an output. “How many jellybeans in the jar” will get a seemingly relevant response because it has seen that “How many” usually leads to a numerical answer, “jar” refers usually to glass kitchen containers. Jellybean is associated with other similar sized objects. But it has also seen other times people asked this question in history, and there are various answers, and various jar sizes. So it may ask you jar size. It sounds like a reasonable answer, but it’s a combination of patterns from history. Does that mean it hallucinated the jellybeans? No, it means the system didn’t have enough information, or the right context, to produce the answer you expected. Humans do this constantly and we have a shared experience that allows us to usually get away with it among ourselves. Most importantly, hallucination implies the AI holds a mental model, a model of meaning and then accidentally subverts that meaning. But there was never a mental model.
Our brains have never before had to deal with the idea of a being that seems to act and exhibit judgement but has no agency, so we mentally ascribe to Generative AI the ability to think and choose and express itself. We make it worse by ascribing verbs like “think”, “reason”, etc. Our language implicitly anthropomorphizes objects constantly. To address a prompt to AI, we often say “I want you to…” which implies “you” exists as an entity. Mostly we know better, but language smuggles in extra meaning. If we name an AI “Alice” we anthropomorphize it even further. “Alice thinks…” “Alice told me”. A better mental model might be a diary. “Dear Diary” is a phrase that anthropomorphizes the diary, but it tends not to snowball because the diary can’t provide complex feedback.
There’s a sense of magical realism that comes with thinking about AI. Personally the best antidote I’ve found for that is adjusting model weights and settings. Nothing makes the reality more obvious than increasing the temperature (ability to choose less mathematically ideal terms for its response) on an AI backend. A slight adjustment will make it respond like a 70s beat poet. I actually find this encouraging. The system isn’t a black box the way you’ve heard. It’s too complicated to follow and not designed for humans to understand, but the principles are pretty clear. If you take nothing else from this, consider: the machine has no judgement. Zero. It won’t tell you when something doesn’t make sense or isn’t true. It will always perform its function, but it can’t steer itself and if you under-specify you may not get what you wanted. It’s a car driving into the river with the steering-wheel locked. You’ve got the key to the steering wheel if you want to steer. Where you point it will always be up to you. You’ve heard by now of “Agents”. People say these can automate whole positions. And that’s kind-of true, but very risky. If you want precision or accuracy, you need a deterministic system. Programmers and Analysts have been building these for decades. Most of their work is maintaining existing systems, patching things, etc. None of that goes away when you build this. If you need guaranteed correctness, use a deterministic system. If you use a LLM instead you are trading certainty for speed and flexibility. You’ll need to verify the result. The system won’t tell you when it’s working from incomplete or misleading patterns because it doesn’t understand meaning at all.
The distinction is not between automation and intelligence. It is between tasks that require memory and tasks that can tolerate loss.
When an instruction can be discarded after execution, it becomes delegatable without continuous oversight.
This shifts the design problem from correctness toward containment: defining boundaries where probabilistic execution is acceptable.