How Secure Is the Code Your AI Writes?

You are three lines into a function, and the assistant finishes your thought. The suggestion arrives in grey, plausible, shaped exactly like the code you were about to write — a database query, a file path, a check on a user token. You press Tab. It compiles. The tests pass. You move on. Somewhere in that grey ghost-text, more often than the launch demo would ever admit, was a security bug that now sits in a commit with your name on it.

This is the part of the AI-coding pitch the videos skip. The tools are genuinely useful and they are getting faster; that isn’t in dispute. What they are not is security-aware, and the evidence for that has stopped being anecdotal. Over the past few years — from NYU, from Stanford, from the code-scanning firms that do this for a living — one finding keeps turning up: something like 40 to 45% of the code these assistants generate carries a known vulnerability, and the polish of the output makes you less likely, not more, to catch it.

None of this means the tools are useless or that you should switch them off. It means the confident autocomplete and the secure autocomplete are not the same feature, and only one of them ships by default. Worth understanding why.

The number that keeps coming back

Start with the measurements, because they are unusually consistent. In 2021, researchers at New York University ran the first large security audit of an AI coding assistant, in a paper drily titled Asleep at the Keyboard. They built 89 scenarios targeting the kinds of bugs on MITRE’s “Top 25” list of dangerous software weaknesses, had GitHub Copilot complete each one many times over, and inspected the 1,689 programs that came out. Around 40% contained a security vulnerability. Worse for the beginner, the single top-ranked suggestion — the one a novice is most likely to accept without thinking — was vulnerable roughly 39% of the time.

Two years later, a Stanford team asked the more human question: what happens to people who use these tools? Their study, Do Users Write More Insecure Code with AI Assistants?, put developers in front of security-sensitive programming tasks, half with an AI assistant and half without. The group with the assistant wrote measurably less secure code on four of the five tasks. That alone would be a footnote. The finding that matters is what came next: the same group was more confident their code was secure. The tool did not just fail to help; it quietly talked them out of checking.

If those studies feel dated — different era, older models — the 2025 numbers are worse, not better. The security firm Veracode ran its 2025 GenAI Code Security Report across more than 100 large language models in Java, JavaScript, Python and C#, asking each to complete tasks with a known safe answer. Across the board, 45% of the generated code introduced a vulnerability from the OWASP Top 10, the industry’s canonical list of the most critical web-application risks. Three separate methods, four years apart, and the needle barely moves off “about two in five.”

Why the machine writes the insecure version

The tempting explanation is that the models are bad at code. They aren’t — they are, if anything, alarmingly fluent. The problem is more specific, and it starts with what they learned from. A language model’s abilities are downstream of its training data, and for code that data is the public internet: GitHub, Stack Overflow, tutorials, forum answers. That corpus is not a curated library of exemplary, hardened software. It is decades of real code written by tired people under deadline, and it is full of the exact vulnerabilities the model now reproduces — SQL queries built by string concatenation, user input rendered straight into a page, credentials hard-coded because it was quicker.

The model does not know any of this is dangerous, because it has no threat model. It is not reasoning about an attacker; it is completing a pattern. Ask for “a function that looks up a user by name” and it will produce the most statistically likely continuation, which is the common way — and the common way, across a decade of training data, is frequently the insecure way. The assistant has no idea where your trust boundaries are, which inputs are attacker-controlled, or that this particular endpoint faces the open web. Those are the facts security depends on, and they live in your head, not in the prompt.

The Stanford study caught this in miniature. On a task about handling file paths, the assistant reliably checked that a path started with the right directory but did not canonicalise it first — so a path with a “../” in it could still climb out of the safe folder. The suggestion looked like a security check. It even was one, sort of — only not the right one, and you would only know that if you already knew the answer. This is the same anti-hype instinct we applied to the myth of the 10x developer: the speed is real, but it is speed at producing plausible output, which is not the same thing as correct output.

The confidence trap

The most under-discussed finding in this whole literature is not the 40% — it is the confidence. A tool that produced obviously bad code would be safe, in a perverse way, because you would distrust it. What these assistants produce is idiomatic code: correctly indented, sensibly named, structured the way a competent colleague would structure it. Fluency reads as competence. And a suggestion that looks like a senior engineer wrote it gets a junior engineer’s review — a glance, a nod, a Tab.

You review your own rough draft more sceptically than you review a polished paragraph handed to you, because the rough draft still looks provisional. AI output never looks provisional. It arrives finished. That is why the Stanford result — less secure code, more confidence — is not a paradox but a direct consequence of the interface. The assistant collapses the pause between “here is an idea” and “here is the answer,” and it is in that pause that a developer normally does the thinking that catches the bug.

Security researchers have a name for the broader effect: automation bias, the well-documented human tendency to over-trust a confident machine and under-apply your own judgement. It is the same reason people drive into lakes following a satnav. The coding version is quieter and more expensive: you accept the suggestion, it works in the happy path, and the vulnerability only exists on the path an attacker will go looking for — which, by definition, your passing tests never tried.

Bigger models didn’t fix it

The industry’s standing answer to almost any shortcoming is “wait for the next model.” On security, that answer has so far not arrived. The most striking line in Veracode’s 2025 report is that as the models got better at writing functionally correct, syntactically clean code, their security performance stayed flat. Larger, newer, more sophisticated — no safer. Whatever the training process is optimising, it rewards code that runs far more than it rewards code that is safe, and the two came apart.

The specifics are not encouraging. In Veracode’s testing the models failed to prevent cross-site scripting about 86% of the time and log injection roughly 88% — two classic, decades-old web bugs that any static analyser flags in seconds. Broken down by language, Java was the worst, failing something like 72% of the security tasks, with Python, C# and JavaScript clustered lower but still landing in the 38-to-45% range. These are not exotic edge cases. They are the bread-and-butter mistakes that secure-coding training has warned about since before the model’s authors were born.

There is a familiar lesson here, and it is the one we keep returning to about why benchmarks mean less than you think. A model can top a coding leaderboard — those scores measure whether the code works — while quietly shipping a vulnerability in nearly half of it, because the leaderboard never asked whether the code was safe. A number that measures correctness is not measuring security, and treating the first as a proxy for the second is exactly the mistake the confidence trap encourages.

The feedback loop nobody ordered

There is a slower, structural risk stacked on top of the immediate one. Georgetown’s Center for Security and Emerging Technology laid it out in its November 2024 report, Cybersecurity Risks of AI-Generated Code, which sorts the danger into three buckets: models that generate insecure code, models that can be manipulated into doing so on purpose, and the downstream effect where all that insecure output flows back into the ecosystem.

That third bucket is the quietly worrying one. AI-written code does not stay in one repository. It gets committed, copied, forked, and posted to the same public forums the next model will train on. Insecure patterns that a model emitted this year become training examples that teach next year’s model to emit them again, with more authority. It is a feedback loop: when a system increasingly learns from its own exhaust, its worst habits calcify rather than wash out, and a vulnerability class that might have died out as developers learned better is instead kept on life support by the tools, propagating at machine speed. The same dynamic hands attackers a lever, as we saw with slopsquatting, where a hallucinated dependency becomes a real supply-chain attack the moment someone registers the name.

To be fair: the industry is responding

It would be reverse-hype to leave it there, because the people who build these tools are not oblivious, and some of the response is genuinely good. GitHub, whose Copilot started this literature, now ships Copilot Autofix: pair the code-generating model with a scanner, and when the scanner finds a vulnerability the same AI proposes the fix. GitHub reports it suggests usable remediations for more than two-thirds of detected vulnerabilities, covers over 90% of alert types in languages such as JavaScript, Java and Python, and cuts median fix times roughly threefold. Using AI to clean up after AI is a little on the nose, but it works, and it is the right shape of answer.

The honest steel-man goes further. Human-written code is also insecure — spectacularly so; the entire static-analysis industry exists because people have shipped these same bugs for thirty years. The models did not invent insecure code. They learned it from us, and then scaled it. And the correctness gains are real: for boilerplate, for tests, for the tedious 80% of a task with no security surface at all, an assistant is a genuine multiplier. Point it at a well-instrumented pipeline — scanners, secret detection, dependency checks, a review that actually happens — and the net effect on security can be positive, because the same automation that writes fast can also catch fast. The tool is not the enemy. The blind trust is.

The bug is the tool’s; the commit is yours

Here is where the pro-consumer sting lands, and it is not really about code quality at all. It is about who carries the risk. When the assistant suggests a vulnerable query and you accept it, the vulnerability is now yours — in your repository, under your name, in your company’s product, exposing your users. The vendor’s terms are careful to make sure of that; you will find no warranty that the generated code is secure, fit for purpose, or even yours to use cleanly, a point we picked apart in whether your AI-generated code is really yours.

So the arrangement, stated plainly, is this: the tool takes the credit for the speed, and the developer takes the liability for the flaw. That is a reasonable deal if you know it is the deal — if you treat the assistant as a fast, fallible junior whose output you are professionally responsible for. It becomes a bad deal the moment the marketing convinces you the output is trustworthy by default, because then you have accepted the risk without pricing it. The gap between “it wrote the code” and “you shipped the code” is where every one of these vulnerabilities lives.

What good looks like

Knowing the mechanism turns a vague unease into a short list of habits. None of this is exotic; it is ordinary security discipline, applied with the extra scepticism the confidence trap demands.

• Treat every suggestion as untrusted input. The assistant is a fast source of candidate code, not a trusted author. Give its output the review you would give a pull request from a stranger, not the nod you give your own finished work.
• Keep the machines that check in the loop. Static analysis, secret-scanning and dependency checks were built for exactly this failure mode and do not suffer from automation bias. If AI is writing more code faster, your scanners matter more, not less — this is the spirit of the NIST Secure Software Development Framework, which cares how software is built regardless of who or what typed it.
• Never accept security-relevant code you can’t explain. Auth, crypto, input handling, file paths, anything touching money or personal data: if you cannot say why the suggestion is safe, you are not qualified to approve it yet. Look it up before you accept, not after the incident.
• Calibrate your trust to the task. Boilerplate and tests are low-risk and high-reward. A payment handler or a permissions check is the opposite. Use the tool eagerly for the former and warily for the latter.
• Assume the polish is hiding, not proving, the work. Idiomatic output earns more scrutiny, not less, precisely because it is the kind your review is trained to wave through.

The tool is fast; the caution is yours

The promise of AI coding is that a capable partner writes half your code while you think about the hard parts. That promise is real, and on a good day the tools deliver it. The unadvertised clause is that the partner has never been taught what “secure” means, cannot see the attacker you are defending against, and will hand you its most confident work in exactly the register that discourages you from checking it. Two in five of those confident suggestions carry a known flaw, and the number has not budged as the models have grown.

That is not a reason to put the tools down. It is a reason to keep your hand on the wheel. “Found means fixed” is a fine slogan, and the fixing genuinely is faster than it used to be — but only for the vulnerabilities someone is still looking for. The assistant will happily write the code. Deciding whether it is safe to ship is the part that was always your job, and it still is.