Who Decides Who the Car Kills?

Some families have a WhatsApp group. Mine has a Discord — with channels, and rules, enforced by my brother Tim. There’s one for time-sensitive alerts, one for funny links, one that is mostly cat memes, and then the one that runs hot: news articles. Officially it’s the politics channel. Unofficially it is the Henson Loudmouths — a 24-hour family battle royale conducted at volume across the gap from London to California, the natural output of a household of verbose talkers and writers who have never knowingly let a take go unchallenged. Tim keeps Mum’s posts filed in the right place and the rest of us roughly honest.

I dropped one of my own pieces into it the other week — the one arguing we should let AI run the government — mostly to start a fight, because starting fights is what the channel is for. Mum promised to read it gradually, as I can’t sit and read it right now, alas. And then Tim replied.

Tim did not read it gradually.

Do you want the Butlerian Jihad? Because that’s how you get the Butlerian Jihad.

For those who didn’t spend their formative years in Dune: the Butlerian Jihad is the bit of backstory where humanity, having handed too much of itself to thinking machines, rises up, smashes every last one, and bans them forever. Thou shalt not make a machine in the likeness of a human mind. The ultimate over-correction — a subject I have form on. But Tim wasn’t really threatening a holy war on the toaster. He was making the sharpest objection anyone’s put to me on this, so I want to take it seriously rather than be clever at it.

His point: any plan that says let’s turn part of the government over to computers does not, in fact, turn it over to computers. It turns it over to the people who program the computers. Even with “AI” — and he put the quote marks in, which I loved — you’re just handing control to whoever can tweak the model. Now, I wouldn’t object if that person was me, he added, but since that’s unlikely I don’t think it’s a good idea.

That’s the whole case against machine governance in four sentences, and it’s correct. So I’m going to agree with him completely — and then show you that his objection, followed all the way down, lands us somewhere much more urgent than he thinks. Somewhere we’re already standing, pretending we’re not.

Let me show my working.

It’s not an “it”

Start with the quote marks, because Tim was more right to put them there than he knew.

The biggest thing people get wrong about “AI” is baked into the grammar. We say the AI decided, the AI took his job, the AI is coming for us — and every one of those smuggles in a lie of agency. There is no it. It is not an it, and it is not a he or a she or a they. It is a tool: a very, very efficient way for humans to do things. It didn’t write my blogs. It doesn’t create images or solve problems. It cannot do anything in the sense that matters, because nobody’s home to do it. It is not sentient. It’s a mirror we built out of everything we’ve ever written down, and it’s very good at holding that mirror up fast.

I said a version of this in The Eye Test — the £23-billion company whose “AI transformation” couldn’t answer a WhatsApp. The AI there wasn’t broken; it was pointed at your wallet, not your eyes. Same lesson, one notch more serious here. The machine has no wants. It points wherever it’s aimed. The only question that has ever mattered is who’s holding it, and where they’ve decided to aim.

Which is why the sentence you hear everywhere — the AI is taking your job — is such a tidy piece of misdirection.

Read that twice, because it changes who you’re angry at, and being angry at the right party is the whole game. There’s no it reaching for your P45. There’s a person — a manager, a board, a shareholder — making a plain decision to swap an expensive human for a cheap machine, then letting the phrase the AI took the jobs carry the blame off to a defendant who can’t be sued, shamed, or voted out. The passive voice is doing the firing.

Hold that — a decision made by a person, dressed as a thing that merely happened — because it’s about to get much darker than a redundancy.

The car on the 680

Here’s where the abstract gets a windscreen.

Everyone knows the trolley problem. Runaway tram, five on one track, one on the other, your hand on the lever. It’s been a philosophy-seminar parlour game for sixty years precisely because it’s safe — a distant, bloodless puzzle you get to be clever about over a pint. And that’s the tell I keep writing about: the comfortable version of a hard question is almost always the wrong one. If the debate feels like fun, you’re probably nowhere near the lever. The whole Umbrella Walkers series is five essays on that one reflex.

So let me bolt the trolley problem to the front of one of Elon’s self-crashing Teslas, doing 80 on the 680, when a tyre blows.

The car has to do something — abstaining is a choice, and usually the worst one. So:

Argue the ethics all night; that’s not the point. The point is the sentence underneath, the coldest true thing in this whole piece:

There is no version of a self-driving car that doesn’t contain an answer to left or right. The answer is in there right now, in a file, written by someone, on a Tuesday, to a deadline. Not a hypothetical for someday — the current situation. We already have machines making choices about who lives and who dies, both intentionally in the crash-logic and unintentionally in a thousand smaller weightings, and every one of those choices sits inside a black box almost nobody has opened.

This isn’t the trolley problem as philosophy. It’s the trolley problem as a product spec. And a product spec has an author, the author has a boss, and the boss has shareholders.

Who’s actually holding the wheel

Follow Tim’s logic — whoever tweaks the model — all the way to the money, and the left or right line writes itself.

Tesla has shareholders. And it doesn’t take much imagination to work out that you cannot sell a car programmed to kill you, its owner, to save a couple of strangers on the hard shoulder. Nobody test-drives the car whose advertised feature is a willingness to sacrifice the driver. So — obviously, inevitably, the moment you accept the car must choose — it gets programmed to kill the random strangers and save the driver. Both outcomes are grim. One of them sells more cars. And the difference between them, the bodies on the verge, is exactly the sort of thing a large company can hide away and manage with press and adverts and some charity donations.

That is not the car deciding. That’s a decision, made by people, on an incentive, laundered through a car. The black box isn’t a mind we can’t understand. It’s a P&L we’re not allowed to see. Point the same machine at a different master and the code inverts — force every manufacturer to write “protect the pedestrian” and they would, overnight, and sell just as many cars, because then it’s a level field and nobody’s advert has to mention it. The technology has no opinion. The ownership has the opinion. The technology just enforces it at 80 miles an hour.

And here’s the part Tim will enjoy, because it’s his own worry turned up to the extreme. We didn’t just build a neutral tool — we trained it on ourselves, on the whole written output of humanity, flaws included. So it has our flaws baked in, and baked in harder than we carry them. You know the old conformity studies — the ones from the same slightly-cavalier era that gave us the Stanford prison experiment: get ten people to insist black is white and the eleventh very often goes along, or at least goes quiet. We do that a bit. A model trained to reproduce humanity’s weighted, published best guess does it to the extreme — it will weight the majority over the minority every single time, because that’s what the training rewards.

Sometimes that’s a good thing. Ask a model whether vaccines turn babies blue and it’ll tell you, all day, that they don’t, because the weighted body of trusted evidence says they don’t. It won’t fall for the article your mate shared, because it doesn’t have the human bug where we trust the source — our mate — over the evidence. That bug is where a lot of vaccine “scepticism” actually comes from, and the machine is immune to it.

Or it won’t be immune. Because whether the model tells you vaccines are safe, or tells you there’s a big movement of sceptics and here are their concerns, depends entirely on how it’s been weighted — by whoever tweaks the model. Same tool, opposite output, one choice made by a person off-stage.

We keep hunting for the it making these calls. There is no it. There’s only ever a who, standing just out of frame.

Tim’s objection, at full strength

Now I owe Tim the strongest version of his own argument, because if I can’t answer it, everything above is just a longer way of agreeing with him.

It has two edges, both sharp. First: if AI is only ever a who in disguise, then “let AI decide” is never actually on the menu — every time you think you’ve handed a decision to the machine, you’ve handed it to the machine’s owner. So my original blog is naïve at best: it dresses “hand the country to whoever controls the model” up as “hand the country to a neutral super-referee.” The referee has a boss. You’ve just moved the throne behind a curtain and told everyone it’s empty. That’s how you get the Butlerian Jihad — not machines rising up, but a handful of people quietly governing everyone through them until the rest notice and burn it all down.

Second, cutting the other way: maybe, for some decisions, the machine genuinely is the better place to put them — precisely because it strips out the human incentive. Watch how a politician actually decides whether to back a vaccine rollout. The optimal move for getting re-elected is to say something waffly — I’ve seen arguments on both sides, we must follow the evidence — tuned to not-quite-scare one crowd and not-quite-enrage the other. That’s a decision optimised for the politician’s survival, not the outcome. A machine weighing the actual evidence removes that grubby little incentive to fudge. So there’s a real case that taking some decisions out of human hands is the point.

Both edges are true. And here’s the thing: they don’t sink my argument. They are my argument. The moment you accept that (a) the machine always has an owner, and (b) some decisions really would be better made by it than by an incentive-warped human — you’ve arrived at a question you cannot answer with a shrug. Which decisions? Written by whom? Weighted how? With what limits, what recourse when it’s wrong? I don’t have those answers. But I am much more comfortable in a world where we sit down and fight them out than in the one we’re in — where the answers are being written, right now, one line of crash-logic at a time, by whoever got there first.

That’s the trap in one move. Not choosing is a choice. Refusing the debate doesn’t leave the decision unmade — it leaves it made by default, in the black box, on whichever incentive is loudest, which is almost always the one with shareholders attached. Burying our heads in the sand doesn’t dodge the thing we fear. It’s the surest way to get it.

When Tim is world leader

Tim has a way of ending these arguments. When the thread’s gone twelve rounds and he’s bored, he plays what the family calls his trump card — deadpan, no heat in it at all:

Well, that’s why, when I’m world leader, I’ll just make it so everyone has to do the sensible thing. Anyway, I’m going to go play video games now. Are you leaving soon?

Translation: we’ve reached an impasse and I’m bored. For years I read it as a cheap move, the board-flip of a man who’d lost and wouldn’t admit it. I was wrong about that, and being wrong about it is most of the point of this essay.

Because under the joke sits a bleak, correct observation about how people change their minds, which is: they don’t — not because you told them to. You can’t fool all of the people all of the time, and, worse, you can’t convince them either. People have to convince themselves. Whether it’s whether their jeans make them look fat, whether we really landed on the moon, or whether someone’s putting something in the water to turn the frogs gay — the one person who cannot deliver the truth that finally lands is the person visibly invested in delivering it. Let them hear the same thing a month later from a stranger — someone with no stake in winning them over — and they’ll take it on board, then come back and explain it to you, the original messenger, as though it were news.

There are layers, and Tim gets all of them. Sometimes people are rational. Sometimes they know they’re being irrational and will cheerfully admit they’re cutting off their nose to spite their face. But the state that should frighten you — the one this whole argument keeps circling — is the third: people who are wrong, and informed by nothing, but certain they’re rational, certain they’re across the facts, and who will not hear it otherwise. At least not from you.

Look at what that does to “let the machine decide,” because it cuts both ways at once. On one side it’s the best case Tim’s second edge could make: the machine is the ultimate stranger with no stake, not running for re-election, no jeans in the game — for the confidently-wrong-but-certain, a verdict from a thing with no skin in it might be the only verdict they can accept. On the other side it’s the whole trap in one image, because the machine only looks like the disinterested stranger. It has an owner. It has a stake — theirs. Mistake the neutral-seeming referee for a neutral one and you’ve done the exact thing that makes the confidently-wrong so dangerous.

And Tim’s trump card — I’ll just make everyone do the sensible thing — is, told straight, the oldest temptation in politics: the benevolent dictator, the philosopher-king, the one wise hand that overrides the irrational crowd for its own good. He’s joking. Half-joking. He isn’t humouring anyone and he isn’t conceding — not his style — he just can’t see past the impasse without someone, eventually, being made to. And “let AI run it” is that same fantasy with a lab coat on: a benevolent dictator you never have to admit is a dictator, because it’s a machine, because it’s neutral, because it’s only following the evidence — right up until you ask who decided what the evidence was.

So there’s only one honest way to build the stranger and keep it disinterested. Not one wise hand. All of ours. Out loud. Which, it turns out, we have done before — for a machine that has been killing us since the day it was invented.

We have done this before. It was called a car.

Cars have always killed people. Not eventually, at scale — from the very first breath of the technology.

The first person ever killed by a car died in 1869, in Parsonstown in the Irish midlands, thrown from an experimental steam carriage her cousins had built, falling under its wheels and breaking her neck. Her name was Mary Ward, she was a naturalist and astronomer, and the machine that killed her was doing about four miles an hour. There were almost no such contraptions anywhere on Earth. The car barely existed, and it had already taken someone. The lethality wasn’t a bug that crept in once there were millions of them. It was in the thing from the start.

Which is the real reply to the bumper sticker. Guns don’t kill people, people do is the “it’s not an it” argument wearing a holster — and on its own terms it’s true. The gun has no will; the car has no wants; the model isn’t sentient. But no sane person gets from guns don’t kill people to therefore hand every toddler a loaded one and let them find out the hard way what it does. The neutrality of the tool is not an argument against governing it. It’s the argument for governing who holds it and how. A lethal tool with no intentions of its own is precisely the kind of thing a society has to make deliberate, collective decisions about — because the tool certainly won’t make them for you, and left alone, the person with the most to gain will.

And here’s the hopeful bit, because I don’t want to leave you in the black box. We took the deadliest everyday machine ever handed to ordinary people, and over about a century we did not smash it — no Butlerian Jihad — and we did not simply let whoever sold the most cars decide how it killed. We built a deal around it. And the purest version of that deal isn’t a ban or a nanny-state crawl. It’s the Autobahn.

Germany is the one wealthy country that looked at the fast, genuinely dangerous, genuinely fun thing and decided — collectively, over decades — to keep it. Around 60% of the Autobahn network still has no fixed speed limit. You can do 250 on a Tuesday and it’s legal and it’s brilliant. But — and this is the entire point — it isn’t universal, and it isn’t free. The keeping of the fun came bundled with the most serious driver-responsibility culture on the planet:

And the result of that bargain: Germany runs roughly 3.3 road deaths per 100,000 people — about a quarter of the American rate — while still letting you drive as fast as you dare on most of the motorway.

That is not the machine being safe. That’s a society that decided, out loud and together, to keep a real danger because it valued what the danger came with — and then made everyone buy into the responsibility that made it survivable. It was a collective decision, made over time, that everyone signed up to. No shareholder wrote it in a black box. Nobody had to smash the cars. They just refused to pretend the danger wasn’t there, and refused to let one company decide the terms in silence.

That is the answer to who decides who the car kills. Not Elon. Not the model. Us — if we choose to have the argument and build the deal, the way Germany built the deal around the Autobahn, instead of leaving the left-or-right line to a P&L and pretending it doesn’t exist.

Because make no mistake about the stakes, and here I’ll let Tim’s dread stand. Humanity is, genuinely, a slow-moving cycle of car crashes that get more dangerous each lap. Our biological evolution simply is not keeping pace with our social, cultural and technological evolution — and that gap is, honestly, the most likely cause of our eventual demise as a species. Every time we’ve built something that outran our wisdom, we’ve paid for the lag in blood before we wrote the deal. The car. The bomb. The feed. This is the next one, and it’s the fastest yet.

The good news — the only news that matters — is that the lag is closeable, because we’ve closed it before, and it closes at the tier where people actually argue with each other. Which is where this started. The reckoning I’m describing — the collective, deliberate, out-loud decision about who writes the kill-logic — is one Westminster isn’t having and Silicon Valley would rather we didn’t. But it is happening. It happened this week, in a Discord channel called #funny, between me and my brother over a blog Mum is getting to gradually. You’d find more honest discussion of it in the Henson Loudmouths than in most green rooms — and a family argument that’s a bit too serious for the room it’s in is exactly the tier where every norm that ever mattered got set. The debate refusing to die in a channel meant for cat memes is, genuinely, the machinery of a social contract turning over for the first time.

So have the argument. Have it badly, in the wrong channel, with your brother who thinks you’re building the Butlerian Jihad. Just don’t let it be settled for you — in a file, on a Tuesday, by someone who wouldn’t object if the person tweaking the model were them. It isn’t going to be them. And it isn’t going to be you.

This started as a thread in a family Discord channel called #funny, in reply to We Should Let AI Run the Government, and turned into the hardest objection I’ve had to answer — so credit to my brother Tim, who put the quote marks around “AI” before I did. The Butlerian Jihad is from Frank Herbert’s Dune*; the conformity research alluded to is the Asch line studies of the 1950s. Mary Ward, killed at Parsonstown (now Birr), Co. Offaly on 31 August 1869, is the first recorded automobile fatality. The Autobahn figures — roughly 60% of the network without a fixed speed limit, and German road deaths of about 3.3 per 100,000, around a quarter of the US rate — are from German road-safety and ADAC/Destatis data; the licence, TÜV and mandatory-equipment rules are current German law. The opinions — and the car on the 680 — are my own.*