GenAI is the greatest magic trick ever performed
Every great magic trick depends on the same thing. Skill matters, and so does the technology concealed in the box or up the sleeve. But what makes a trick work is the audience’s willingness to experience something they know, on some level, cannot be true. The magician does not deceive you. You deceive yourself, gratefully, because the alternative is less interesting.
This is worth sitting with when we talk about generative AI.
The technical foundations of a large language model are neither mysterious nor, at this point, particularly secret. Transformer architectures, attention mechanisms, vast quantities of training data, extraordinary amounts of compute. The researchers who built these systems have published their methods. The magic is, in principle, available for inspection. And yet the dominant public response has been to treat the outputs as evidence of something more than the mechanism: to conclude that because the text is fluent, the thing producing the text must be thinking; that because the answers are often right, something must understand the questions.
Neither of these follows. The coin did not disappear.
What is more interesting to me than the trick itself is what it reveals about the audience. The desire to believe that we have conjured a new kind of mind is entirely rational. Framing GenAI as a new kind of mind changes everything. Framing it as a remarkable but ultimately explicable achievement in mathematics and computing does not. People have always preferred the more dramatic framing, and there is nothing uniquely foolish about preferring it now.
But preferences have consequences, particularly when they shape institutional decisions.
My own research sits at the intersection of cryptography, cybersecurity, and increasingly, AI security. From that vantage point, what looks like wonder from the outside looks rather different. AI systems, including the generative kind, fail in ways that are not intuitive, and they fail confidently. Feed a well-crafted adversarial input, a perturbation invisible to the human eye, into an image classifier, and it will misidentify what it sees with complete assurance. Prompt a language model in certain ways and it behaves in ways its designers neither intended nor anticipated. These show up well outside academic red-teaming exercises. They are structural properties of how these systems work, what incentives shape the output, and under what conditions the confidence of that output bears any relationship to its accuracy.
Organisations that deploy these systems without working through the failure modes, the training data provenance, and the conditions under which confident output becomes unreliable output, have not adopted a technology. They have acquired a dependency on something they cannot diagnose when it goes wrong, and cannot fix without going back to the people who built it. In security terms, this is exactly the position I’ve spent my career trying to avoid placing myself in.
None of this is an argument for scepticism as a posture. The technology is genuinely impressive, and the people building it are doing serious work. The performance around it, the language of civilisational transformation, the carefully staged demonstrations, the relentless sense of urgency, that is worth keeping distinct from the underlying science. The history of technology is full of moments where the story outran what the technology actually did, and where the gap between them caused real damage. The question worth carrying is not whether the technology is impressive, but how it actually works, where it fails, and what you are really handing over when you trust it.
Three examples from recent decades illustrate this pattern: 1) the dotcom bubble of the late 1990s, in which internet valuations detached entirely from business fundamentals before collapsing; 2) the blockchain wave of 2017–18, which promised to disintermediate practically every institution before entering what analysts called a "blockchain winter"; 3) and the self-driving car decade, in which virtually every major manufacturer predicted full autonomy by 2020–21, a prediction that proved comprehensively wrong.
Read our papers
Tuovinen, Lauri, and Kimmo Halunen. “What Is an AI Vulnerability, and Why Should We Care? Unpacking the Relationship Between AI Security and AI Ethics.” Proceedings of the Conference on Technology Ethics 2024 (Tethics 2024), CEUR Workshop Proceedings, vol. 3901, 2024, pp. 30–41.
Pispa, Arttu, and Kimmo Halunen. “Comprehensive Artificial Intelligence Vulnerability Taxonomy.” Proceedings of the 23rd European Conference on Cyber Warfare and Security. Academic conferences international, 2024.
Professor