AI security fixes need verification, not just successful execution

What they studied

The authors reviewed 53 peer-reviewed studies published between 2023 and 2026 on language models and agentic AI for vulnerability management. Their search followed PRISMA 2020 guidance and covered six scholarly databases, with additional update and citation searches.

The review grouped the research into four areas: finding vulnerabilities; enriching and prioritizing vulnerability information; investigating threats and exploitation; and repairing vulnerabilities, responding to incidents, and carrying out autonomous defence. For people building software with AI coding tools, the most directly relevant parts concern detecting security flaws and checking automated repairs.

What they found

Model-focused studies provided the most standardized benchmark evidence. However, reported performance depended on how datasets were constructed, the balance of examples, data leakage, available code context, programming-language coverage, and whether results transferred across datasets. A benchmark score therefore needs to be read alongside the conditions that produced it.

More recent work increasingly combined models with retrieval, structured program or security knowledge, external tools, multi-agent coordination, and verification. Yet only 15 of the 53 studies, or 28.3%, implemented or directly evaluated systems that met the review’s definition of genuinely agentic behaviour. That definition included adaptive evidence gathering, iterative tool use, feedback-driven control, coordination, or bounded action.

The authors also distinguished capability demonstrations from operational readiness. Exploit and autonomous-defence evaluations did not directly establish autonomous vulnerability-management capability. In repair and response studies, successful execution alone did not establish that a vulnerability was removed, that program behaviour remained correct, or that a change was safe and production-ready.

Why it matters for builders

If an AI coding tool proposes a security fix, treat “it ran successfully” as a limited result—not proof that the security problem is solved. The review supports checking vulnerability removal, intended behaviour, and operational safety separately.

Its broader conclusion favours bounded, auditable AI assistance within workflows grounded in evidence. For builders, that means using AI to assist security work while keeping its actions constrained and its proposed changes open to verification, rather than treating autonomous execution as a substitute for security assurance.

Source: "Large Language Models and Agentic AI for Vulnerability Management: A Systematic Review" (2026-09-28, Expert Systems). This is our summary of the paper's abstract — read the original for full details.