The most common objection to letting AI write production code is some flavor of this:
“But you can’t prove it’s safe. You can’t prove it’s correct. A human would understand the system; the model is just guessing.”
It’s a fair instinct. It’s also, on closer inspection, a fully general objection — one that, applied honestly, disqualifies nearly every piece of software you already trust with your money, your messages, and your medical records.
Because here’s the uncomfortable truth the AI debate keeps stepping around: software was never black and white. It was always stochastic. We just rounded the probability up to 1 and called it “verified.” AI didn’t introduce the uncertainty. It dragged it into the light.
Let me show you what I mean with the two systems people reach for when they want to argue the opposite — the two gold standards of “we did it the rigorous way.”
Myth #1: “But formally verified systems are provably safe” Link to heading
If any software on earth deserves the word proven, it’s a national election system.
Switzerland’s online voting system — Swiss Post e-voting — is about as close to the platonic ideal of trustworthy software as we get. Cryptographic protocol with mathematical proofs of verifiability. Source code published for public scrutiny. Ongoing intrusion tests and bug bounties. Reviewed by some of the best cryptographers and election-security researchers in the world. This is not a startup shipping on Friday afternoon. This is the careful, expensive, do-it-by-the-book end of the spectrum.
And in 2019, when the source was opened for public examination, researchers found a trapdoor in the verifiability proof — a flaw in the implementation of the cryptographic shuffle that could, in principle, allow undetectable manipulation of the result. The exact thing the math was supposed to make impossible. The system was pulled. Online voting in Switzerland was suspended.
It took a second iteration — a substantially rebuilt system, fully open-sourced, re-audited, re-attacked — before online voting cautiously resumed in a handful of cantons in 2023.
Here’s the part that matters most, and it’s not the bug. It’s that even the rebuilt, relaunched, formally-verified system isn’t “100% safe” — and never claimed to be. The proofs hold under explicit trust assumptions that the formalism deliberately puts out of scope: that the setup components are honest, that the print office is trustworthy, that the voter actually checks their return codes, that the voter’s own device isn’t compromised. The math is real and valuable. But it’s a floor on confidence under stated assumptions, not a guarantee about the world.
So the most-verified software on the planet is something like 99% — a staggering, hard-won 99% — but not 100%. It took two turns to get the cryptography right, and a whole category of risk lives outside the formalism by design.
If “you can’t prove it’s perfectly safe” disqualifies AI, it disqualifies this too. (I hacked it anyway, but the mathematical proof held.)
Myth #2: “But open source is secure — many eyes review everything” Link to heading
The other comfort blanket is Linus’s Law: “given enough eyeballs, all bugs are shallow.” Open the source, the reasoning goes, and the crowd will catch what any individual would miss. Openness is security.
I’d like to introduce a counter-anecdote I found myself.
A few weeks ago I went looking through OpenProject — a popular, widely-deployed, fully open-source project-management tool. I found a pre-authentication remote code execution vulnerability rated CVSS 9.9 (CVE-2026-46386). About as critical as it gets: an unauthenticated attacker could self-register and run arbitrary code as the application process.
The root cause is almost comically mundane. The official Docker image shipped with a hardcoded default Rails secret: SECRET_KEY_BASE=OVERWRITE_ME. Combined with Ruby’s Marshal cookie serializer, anyone who knew that default — which is to say, anyone who read the public repository — could forge a signed cookie carrying a malicious serialized object and get it deserialized into code execution.
Now sit with the timeline. That default was introduced on January 14, 2019. The vulnerability lived in shipping config for roughly seven years, in a popular open-source codebase, in plain sight. A literal placeholder string named OVERWRITE_ME — practically a sticky note reading “replace this or you’re owned” — sat there for seven years while “everyone could review it.”
That’s the thing about many eyes: it describes a potential, not an event. “Everyone can review the code” is true and meaningful. “Everyone did review the code, carefully, adversarially, all the way down to the cookie serializer” is a fantasy. Eyeballs that could look are not the same as attention that did look. Openness is necessary-ish. It is nowhere near sufficient. Review doesn’t happen for free, and the boring, seven-year-old corners are exactly where nobody is looking.
The synthesis: it was always a probability Link to heading
Put the two myths side by side and they collapse into the same shape.
Formal verification promised certainty and delivered high confidence under assumptions, after two tries. Open source promised security through scrutiny and delivered the possibility of scrutiny that mostly didn’t happen. Both are comfort blankets thrown over the same truth:
Software safety is a probability distribution you push toward 1 and never actually reach.
This was always true. It was true before any of us typed a prompt into a coding agent. The compiler had bugs. The dependency had a backdoor. The “thoroughly reviewed” PR shipped the regression. We’ve simply been trained to express our actual confidence — 95%, 99%, 99.9% — in the binary vocabulary of “tested,” “audited,” “verified,” “secure.” The words are categorical. The reality underneath was always continuous.
Which means the honest question was never “can you prove it’s safe?” — because nothing ever cleared that bar, not the election system, not the kernel, not your bank. The honest question is:
What’s our estimated safety, and how do we raise it without bankrupting ourselves?
AI doesn’t change that question. It just makes it impossible to keep hiding behind the categorical words, because now there’s an obvious, named source of stochasticity sitting in the room — and suddenly everyone wants to talk about probabilities they were happy to ignore when the stochastic process was a tired human at 6pm.
Why AI actually moves the number Link to heading
Here’s the turn, and it’s the optimistic one.
The OpenProject bug survived seven years for a structural reason: deep, adversarial, boring review is scarce and expensive. Reading a cookie serializer for the hundredth time, asking “what’s the default secret and what happens if nobody overwrites it,” tracing a deserialization gadget chain — this is exhausting, low-status, easy-to-defer work. Humans don’t do enough of it because there’s never enough time, and the line of code that owns you is rarely the line anyone wanted to read.
This is precisely the economics AI inverts.
A multi-agent review pipeline doesn’t get bored. It doesn’t skip the seven-year-old config because it looks settled. You can run a panel of skeptical, adversarial reviewers — each with a different lens: deserialization, auth boundaries, supply chain, secrets management — against every PR, every dependency bump, and every forgotten corner of the codebase, on every commit, for a cost that rounds to nothing compared to an engineer’s afternoon. The thing that actually raises estimated safety — attention applied to code — goes from scarce to abundant.
This is the bulk of what I’ve been building and running:
- Multi-agent security review that fans out adversarial reviewers across a diff, then has independent agents try to refute each finding before it’s reported — so the signal that reaches me is high-confidence, not a wall of plausible-sounding noise.
- Standing audit passes in the spirit of a tireless CSO: secrets archaeology, deserialization and injection sweeps, dependency checks — the boring archaeology that finds the
OVERWRITE_MEbefore an attacker does. (And yes — point some of those agents at your supply chain too. Half of “your” code is other people’s containers and packages, and that’s exactly the kind of boring, never-reviewed corner where the next seven-year-old bug is hiding.)
None of this hits 100%. Nothing hits 100% — that was the whole point. But you don’t need 100%. You need to reliably move a system from the maybe-80% that realistic human attention delivers up to a defensible 98 or 99% — and to do it cheaply enough that you actually do it, on every change, instead of once a year if a budget appears. That’s a number you reach not with a single oracle that “proves” correctness, but with layers of independent, fallible, adversarial reviewers whose errors don’t correlate. Probabilistic components, composed into a system whose aggregate confidence is far higher than any single part. Which, if you squint, is exactly how the Swiss cryptographers got to their 99%: not one proof, but many independent checks under stated assumptions.
Stop demanding a proof nobody ever had Link to heading
So when someone says “I won’t trust AI-written code because you can’t prove it’s safe,” the right response isn’t to argue that the model is smarter than they think. It’s to point out that the standard they’re invoking has never been met by anything — not by the formally-verified election system that needed two tries and still trusts the print office, and certainly not by the open-source tool whose default secret was OVERWRITE_ME for seven years.
Safety was always stochastic. The professional move was always the same: estimate your confidence honestly, find the cheapest lever that raises it, and pull that lever relentlessly. For decades the cheapest lever — thorough, adversarial, never-bored review — was also the most expensive to staff, so we pulled it sparingly and dressed up the result in categorical language.
AI didn’t break that bargain. It made the lever cheap.
The uncomfortable truth was always there. The comfortable part is what comes next: for the first time, we can actually afford to look.