This is an edited transcript from the third evening of the Harbourside Festival of Ideas in Falmouth, where an audience of about four hundred sixth-formers and teachers filled the hall. The chair, Ruth Penhaligon, was joined by Dr Anselm Wray, a computer scientist who spent nineteen years building systems that help doctors read X-rays, and Professor Leonie Castell, a philosopher who writes about technology and attention.
Ruth: Let me begin bluntly. Should we be afraid? Anselm: Afraid is the wrong word, though I understand why people reach for it. Fear is a poor counsellor but an excellent alarm. What I argue for is proportionate unease: enough worry to make us test things carefully, not so much that we freeze. Leonie: I'd go further. I think fear actually flatters the machines. It casts them as protagonists in a drama, when they are, for the moment, very elaborate tools. The question I find more pressing is what happens to us when we hand over our judgement without noticing.
Ruth: Anselm, what keeps you uneasy? Anselm: Opacity, mainly. Early in my career we built a program that seemed uncannily good at spotting cows in photographs. It turned out to be spotting grass. Every cow it had been shown was standing in a field, so it had learned the field, not the animal. Nobody intended that; the system simply found the easiest route to a high score. Now scale that up to systems that advise on loans or medicine, and you see the difficulty: a confident answer can rest on a foundation nobody has inspected. Leonie: That's a lovely story, and it rather proves my point. The machine didn't conspire against anyone. The failure was ours. We mistook a high score for understanding.
Anselm: Partly. But you can't inspect your way out of every problem when the systems are far too large for any one person to read. That's why I'm wary of the cheerful line that it's 'just a tool'. A hammer doesn't surprise you. Leonie: Granted. Yet notice what the word 'fear' does to an audience. It makes people either apocalyptic or dismissive, and neither mood is good for thinking. My grandfather was a navigator on trawlers out of Brixham. When satellite navigation arrived, he kept drawing his own charts by hand, and the younger crew teased him for it. His reply was that a chart you have drawn is a chart you remember. He wasn't afraid of the new equipment; he used it every day. He simply refused to let it do his noticing for him.
Ruth: So your worry is complacency rather than catastrophe? Leonie: Precisely. The danger is not that machines will begin to think like us, but that we may quietly stop thinking for ourselves. Deference is comfortable. It arrives disguised as convenience. Anselm: I don't disagree, but I'd resist the idea that it's one or the other. Watching our own habits and watching the systems are not rival projects. A pilot trusts the autopilot and still trains for the day it fails. That, to me, is the right posture: trust, but rehearsed scepticism.
Ruth: A student in the front row has a question for you both. Priya: I'm seventeen, and I'm studying computing. If we're not supposed to be afraid, why do the people who build these systems sometimes sound the most worried? Anselm: Because we've seen the cows-and-grass moments up close. Familiarity breeds not contempt but a certain humility. I'd be far more alarmed if the builders sounded entirely relaxed. Leonie: And, if I may be slightly mischievous, worry can also be a kind of advertisement. To say 'what we have made is so powerful that it frightens us' is not, commercially, the worst sentence in the world. I don't say that's cynical in every case, only that we should listen to warnings with the same care we give to promises.
Ruth: We're nearly out of time. Is there anything you agree on? Anselm: More than the format suggests. We both think the deciding factor is human judgement, and whether it's exercised or abdicated. Leonie: And we both think schools matter more than science fiction. If I could change one thing, it would be to teach young people to ask of any answer, human or machine, 'How do you know?' Anselm: I'd only add: 'And how would we find out if you were wrong?' Ruth: On that note of shared, if unequal, unease, thank you both.