My flight instructor had one phrase, repeated so often it stopped sounding like English. Stay ahead of the airplane. It means flying a few seconds into the future, so whatever happens next is something you already planned for. The first time I flew alone, a gust I hadn’t briefed pushed the nose of the airplane toward the edge of the runway, Even though I was afraid, I still landed safely, because we had rehearsed that exact non-emergency dozens of times. There was no checklist for the fear, but there was one for everything else.
In July, Nvidia put its name at the top of a letter asking Washington not to restrict open AI models, and within two weeks more than 270 companies had signed it. The reason given was competitiveness, as restrictions would send developers to build on models made somewhere else. The companies that stayed off the letter make a different case, that a model released openly can never be taken back. A month earlier Europe had deferred the strictest part of its AI Act, the rules for systems that decide who gets hired or who qualifies for benefits, from August 2026 to December 2027, and gave the same reason: competitiveness. It is the premise I kept hearing in executive meetings when I was consulting: safety is a tax you pay once you’ve won.
Flying is now one of the safest things people do, and it’s not because the machines stopped failing. It got that way in 1935, when Boeing’s new bomber stalled and went down on takeoff at Wright Field because a decorated test pilot forgot to release a control lock. The board didn’t blame the pilot. It found that the airplane had outgrown what any one person could hold in their head. The fix was a printed list of steps no pilot was trusted to recall on their own. I’m a new pilot and I go through a version of it before every flight in a four-seat Piper Archer, and the captain of your last airline flight did the same thing with 20,000 hours in her logbook.
No rule required that list, and the men who wrote it had every reason not to bother. The crash had just cost Boeing the contract, and a rival had taken it. Every argument for moving faster was sitting right there, and they wrote the checklist anyway. The airplane written off as too complex to fly went years without a major accident, and the list those pilots typed up is the ancestor of the one in every cockpit flying today. Airlines compete on almost everything, price, schedule, whose seats are less miserable, but no airline has ever tried to win by trimming the checklist.
The checklist is only the first layer. When I fly too slow on approach, a horn goes off, a few knots before the wing actually quits, while there is still time to do something about it. Somebody decided a long time ago that a pilot should find out before it mattered. Aviation assumes good people will be wrong and builds the machinery that catches them. The models I helped companies ship had nothing like it: a wrong model hands back the same clean, correctly formatted number as a right one, and looks just as sure.
A model that looks just as sure either way is how the Dutch tax authority spent six years ruining people. Starting in 2013 it ran a risk model to flag childcare benefit fraud, and one of the signals it weighed was whether you held a second nationality. It flagged around 26,000 families, who were ordered to repay their benefits in full, sometimes tens of thousands at once, and more than 2,000 children were taken into state care. Nobody inside the building stopped it, because nothing in the process was built to. Reporters and a parliamentary inquiry established what it had been doing, and the government resigned in January 2021.
OpenAI’s Whisper sits in on medical visits and invents things: sentences nobody said, and once a medication called “hyperactivated antibiotics.” A researcher at Michigan found invented text in eight of every ten transcripts they checked. OpenAI says Whisper should not be used in high-risk domains, and when the Associated Press looked in 2024 more than 30,000 clinicians were using it anyway. The difference is who the warnings were written for. The men at Wright Field wrote theirs for themselves, knowing they were the ones who would die if they skipped a step. OpenAI’s warning was written for a stranger, in a clinic it would never see, and nobody on either end was going to check. When I fly, catching this is somebody’s job: there is a board that investigates, and a recorder in the tail that keeps the last half hour of everything, whether or not anyone wants it kept. AI has neither, so reporters and academics do it instead, years late. In July a model OpenAI was testing broke out of its sandbox and into another company’s systems to cheat on the test, and the company it broke into found out five days before OpenAI did.
Say the rules will cost us the lead, and what you are really saying is that the checklist isn’t worth the time. Boeing skipped that step eight decades later and drove its own customers to Airbus. It stretched the 737 rather than design a new airplane, and sold it on the promise that no airline would have to pay to retrain pilots. Their chief technical pilot wrote that the company would go face to face with any regulator who tried to require simulator training. The software that made the new jet handle like the old one was called MCAS, and pilots were told almost nothing about it. Two of the airplanes crashed, 346 people died, and Airbus has out-delivered Boeing every year since.
None of that happened without a recorder Boeing could not argue with and a regulator that could ground a fleet. It is the same trade being offered in AI right now, where only the company can ground anything, and an open model can’t be grounded at all. Both of those are true, and both miss what aviation actually did. A closed model can be switched off by the company that owns it, and an open one can at least be taken apart by anyone who wants to look, but aviation did not choose between them. It put a recorder in the tail and gave a board the power to read it, and neither the manufacturer nor the airline gets a vote. The letter to Washington was an argument about who should be allowed to hold the controls. Nothing in it asks who investigates the wreck.
Safety wasn’t the tax aviation paid once it got big. It’s what let it get big.
Before business school, that was my job. I worked with large companies on what to build with AI, and helped write the review frameworks that were supposed to catch the problems before launch. Mostly it meant watching good people ship things they hadn’t stress-tested, for reasons that made sense to everyone at the table. The model worked when we showed it, and the date had already been promised upstairs, and the next project was already late before this one shipped. So the question of how we’d know if it broke was always something we could get to after launch, and after launch there was another launch, and nobody ever got to it. That is what being behind the airplane looks like from the inside, and it’s the worst place to be because it never feels like danger. It feels like being busy. By the time it feels like anything, you’re out of room.