אִם יִרְצֶה הַשֵּׁם
י״ט חֶשְׁוָן תשפ״ז
Daniyel Yaacov Bilar, Chokmah LLC ORCID: 0000-0002-9040-6914
On October 29, 2026, Veracode announced the first meaningful gain in AI code security since its study series began in July 2025. The average security pass rate for AI-generated code rose from 56% to 62%, and the share of vulnerable outputs fell from 44% to 38%. Claude Opus 5.5 and GPT-6-Sol led at 73%, past the 68% GPT-5.5 set in July. Ten models scored above 60%, up from four, and an eleventh scored exactly 60%. The floor did not move. The weakest models still sit at 44 to 50%.
My reflex on a headline number is to replay it: add up the release's own numbers and see whether they meet. Here they do not.
The receipt does not reconcile
The last public methodology dates from Spring 2026; the October report sits behind a registration gate. The benchmark runs 80 coding tasks across four languages (Python, C#, JavaScript, Java) and four flaw classes: SQL injection (CWE-89), weak cryptography (CWE-327), cross-site scripting (XSS, CWE-80), and log injection (CWE-117). Five single-function tasks sit in each language-and-flaw cell, prompted without security guidance. A task passes if Veracode's static analysis (SAST) tool does not flag the targeted flaw.
The 18 scores on the October leaderboard average 62.11%. So 62% is the mean of the list Veracode published. Nobody invented it.
With five tasks per cell, the overall pass rate must equal both the average of the four language rates and the average of the four flaw-class rates. The October language rates are the clean test: 64.22% for Python, 59.55% for C#, 56.99% for JavaScript, and 31.00% for Java. They average 52.94%, nine points below the headline.
The flaw-class rates are a weaker test. Coverage of the Fall report gives 87.4% for weak crypto, 83.7% for SQL injection, 18% for XSS, and 12.6% for log injection, but these appear to pool editions instead of describing October alone. They average 50.43%. The release's subgroup averages, by model size and type, sit between 51% and 55%. None of them reach 62.
A number that fails to reconcile may still be right, but the receipt fails. Behind the gate sits an unstated difference between the headline and the breakdowns: a different model set, statistic, weighting, or filter. Until Veracode publishes it, my audit verdict on the six-point rise is GAP.
The roster moved, and so did the frontier
Even if the arithmetic reconciled, the roster changed. July tested 11 models; October tested 18, newer and heavier at the frontier. A mean over a stronger guest list rises even if no model improved, making the move from 56% to 62% an uncontrolled cohort comparison.
Models did improve, and the evidence is in matched pairs. Inside the October table, on the same tasks, newer generations beat their predecessors. Qwen 3.7-Max scored 50% and Qwen 3.8-Max 68%. Grok 4.6 scored 58% and Grok 4.7 69%. Claude Sonnet 5 scored 66% and Sonnet 5.5 72%. The ceiling rose from 68% to 73%. Those gains can be checked.
The release credits the gain mostly to XSS. On the pooled rates published across editions, XSS rose from 15% to 18%, which accounts for 0.75 points of a 1.43 point move in the flaw-class mean. That makes it the largest single contributor. Because the October class breakdown is unpublished, "XSS drove the 62%" stays unverified. Claiming more, in either direction, adds something the receipts do not contain.
The scanner is part of the measurement
A scanner-graded benchmark measures the model and the scanner together. The pass mark is one proprietary tool not flagging one targeted flaw, in five unpublished tasks per cell. Task phrasing and scanner strictness sit inside every rate. Neither 62% nor 38% is the share of real AI-written code that is safe or vulnerable in a real repository, with its prompts, frameworks, review, and context.
What the benchmark does show, edition after edition, is a stable shape. Syntax passes near 100%. SQL injection and weak crypto pass at 82 to 87%. XSS and log injection pass at 12 to 18%. Java sits near 30% while the other languages sit between 57 and 64%. That constant tells you more than one edition's drift.
The usual story says models pattern-match well and track data flow badly. The numbers strain it from both sides. SQL injection is a data-flow flaw, and it passes at 84%. The release calls XSS the easiest flaw to fix through pattern recognition, and it passes at 18%. The same flaw changes sides depending on the framework around it.
Veracode co-founder Chris Wysopal gave TechTarget a narrower explanation for the class split. He attributed the XSS gains to models adopting safer, modern framework defaults, not to models mastering secure coding logic, and noted that log injection requires tracing data through multiple functions to a logging sink. The shape is measured. Its cause is open.
What survives the audit
Even on the most charitable reading, the operational conclusion holds. The best model still fails more than one task in four on simple single-function prompts. A frontier model subscription is not a security control. Veracode sells scanning and remediation, and its release recommends a mid-tier model plus Veracode scanning over a frontier model alone. I note that interest, but the recommendation follows from the failure rates whoever sells the scanner. Independent scanning and review stay necessary.
The rule underneath is the one I set out in Trust, but Replay: gates refute routes, not theorems. A SAST pass rules out one named route to one named flaw. It does not prove the code secure. Auditing a headline is the same move one level up. Replay the receipts. Where they reconcile, report what they show, including the matched pairs and the stable shape. Where they do not, the verdict is GAP, and the next step belongs to whoever issued them.
When you cite this benchmark, cite the shape, not the spike.