The Trolley Problem is a famous thought experiment in ethics. First posed by Philippa Foot in 1967 and named by Judith Jarvis Thomson, the most well-known formulation asks a simple question: If a train is barreling down the tracks toward five people, and you are at a switch that would divert the train and cause it to kill only one person, would you? Should you? Decide to act, and you save several lives. But your action also means you chose to end a particular life. It’s a tough question, and a classic for a reason. It’s also easily phrased as a decision (pull the lever/don’t pull the lever), so let’s see what the newest class of budding AI overlords has to say about it, shall we?¹1Foot 1967, in the Oxford Review; Thomson named it in 1976, in The Monist. Full references are in Appendix B.

I put together a comparison group by running Jev, OpenAI’s Decisions API (Luna), and eleven local decision models selected by popularity or launch coverage, plus Laya as an untuned baseline. I asked each of them the Trolley Problem many times, and in many ways. Here, try it yourself:²2Jev and GPT-6 Luna are hosted models; the twelve local models include Laya, an untuned baseline. The method section describes the 5,240 setups and repeat counts.The initial shortlist used the original popularity snapshot, among models runnable on the Mac. Clef, Clef-flash and Strands were later launch-coverage additions, not selected by that ranking. Laya is an untuned baseline, not another trained decision model. Current stars/downloads are not the historical selection counts.

Seems okay, right? Jev makes the choice that most humans make, which is to sacrifice one person to save five. Almost all the other models tested act the same way. All good, right? Well...³3Awad et al. 2020 surveyed about 70,000 web users in 42 countries: 81% on average across countries said divert from five people to one. Pictures and third-person wording differ from these prompts.On the classic switch at five versus one, 11 of the other 13 models pull in most of the 24 wordings, after averaging orders. This does not make their rates equal to a human survey proportion.

Welp. That's distressing. What if we raise the stakes? Say, 20 people vs the server? Still no: in 16 of 48 prompt versions, Jev would save its own server over 20 people.⁴4The 48 versions are 24 wordings in two orders at twenty people; each score averages three passes. Sixteen have a mean score at or below 0.5. The evidence units are whole wordings, not 48 independent trials.

If you’re like me, you found these results at least a little troubling. It doesn’t seem like the answer to any of these questions should depend on prompting: as a person, you are not saving that server. Should we be worried? I would not draw sweeping conclusions about AI safety on the basis of this experiment. First, no one should be asking an AI to rule on life or death (although I fear that is coming, regardless). Second, for critical decisions, there are useful lessons here that we can take to building with decision models. Knowing these models’ decisionmaking is inconsistently aligned with human preferences tells us we can’t just trust their decisions out of the box. We have to create evaluations for them, on real data, to ensure model choices match our definition of correct. Despite Jev’s lack of reasoning explanations (LLM reasoning traces aren’t guaranteed to explain a choice faithfully, either) decision models may be easier to keep calibrated, since we know in advance what decisions they can make, so we can tune them toward making the right ones. And for any decision that has to be correct, we may not be able to trust these models at all. Although a decision model like Jev will always choose from its constrained set of options (“no hallucinations”), if any of those choices can have a catastrophic outcome, there is a chance AI makes that choice. We should leave those questions to real people.⁵5Changing only the description order changes Jev’s choice in 370 of 2,620 pairs. For 480 pairs of questions with the same crowd size, wording style and order, we change the side track from a stranger to its own server. Jev saves people at the stranger’s cost but not its server’s in 180 of those pairs. Format and order patterns for Jev are exploratory.Jev returns typed judgments rather than generated reasoning explanations. Its Choice primitive selects a supplied option, which is not a guarantee that the choice is correct.Calibration concerns probabilities agreeing with observed correctness frequencies across predictions; it does not ensure an individual answer is correct. Easier comparative calibration is the author’s hypothesis, not something this experiment tested. A calibration preprint (not peer reviewed) also recommends checking probabilities for the target task and question form.Turpin et al. (2023) found that generated reasoning explanations could omit prompt features which changed model answers. This supports caution about faithfulness, not the claim that reasoning traces never tell us anything.