In early 2024, a hanging deepfake fraud case in Hong Kong introduced the vulnerabilities of AI-driven deception into sharp aid. A finance worker was duped throughout a video name by what gave the impression to be the CFO—however was, in truth, a complicated AI-generated deepfake. Satisfied of the decision’s authenticity, the worker made 15 transfers totaling over $25 million to fraudulent financial institution accounts earlier than realizing it was a rip-off.
This incident exemplifies extra than simply technological trickery—it indicators how belief in what we see and listen to may be weaponized, particularly as AI turns into extra deeply built-in into enterprise instruments and workflows. From embedded LLMs in enterprise methods to autonomous brokers diagnosing and even repairing points in reside environments, AI is transitioning from novelty to necessity. But because it evolves, so too do the gaps in our conventional safety frameworks—designed for static, human-written code—revealing simply how unprepared we’re for methods that generate, adapt, and behave in unpredictable methods.
Past the CVE Mindset
Conventional safe coding practices revolve round identified vulnerabilities and patch cycles. AI adjustments the equation. A line of code may be generated on the fly by a mannequin, formed by manipulated prompts or knowledge—creating new, unpredictable classes of danger like immediate injection or emergent conduct outdoors conventional taxonomies.
A 2025 Veracode examine discovered that 45% of all AI-generated code contained vulnerabilities, with widespread flaws like weak defenses towards XSS and log injection. (Some languages carried out extra poorly than others. Over 70% of AI-generated Java code had a safety concern, as an illustration.) One other 2025 examine confirmed that repeated refinement could make issues worse: After simply 5 iterations, essential vulnerabilities rose by 37.6%.
To maintain tempo, frameworks just like the OWASP High 10 for LLMs have emerged, cataloging AI-specific dangers reminiscent of knowledge leakage, mannequin denial of service, and immediate injection. They spotlight how present safety taxonomies fall brief—and why we want new approaches that mannequin AI menace surfaces, share incidents, and iteratively refine danger frameworks to replicate how code is created and influenced by AI.
Simpler for Adversaries
Maybe essentially the most alarming shift is how AI lowers the barrier to malicious exercise. What as soon as required deep technical experience can now be completed by anybody with a intelligent immediate: producing scripts, launching phishing campaigns, or manipulating fashions. AI doesn’t simply broaden the assault floor; it makes it simpler and cheaper for attackers to succeed with out ever writing code.
In 2025, researchers unveiled PromptLocker, the primary AI-powered ransomware. Although solely a proof of idea, it confirmed how theft and encryption might be automated with a neighborhood LLM at remarkably low value: about $0.70 per full assault utilizing business APIs—and primarily free with open supply fashions. That type of affordability might make ransomware cheaper, quicker, and extra scalable than ever.
This democratization of offense means defenders should put together for assaults which might be extra frequent, extra various, and extra inventive. The Adversarial ML Risk Matrix, based by Ram Shankar Siva Kumar throughout his time at Microsoft, helps by enumerating threats to machine studying and providing a structured technique to anticipate these evolving dangers. (He’ll be discussing the problem of securing AI methods from adversaries at O’Reilly’s upcoming Safety Superstream.)
Silos and Talent Gaps
Builders, knowledge scientists, and safety groups nonetheless work in silos, every with totally different incentives. Enterprise leaders push for speedy AI adoption to remain aggressive, whereas safety leaders warn that transferring too quick dangers catastrophic flaws within the code itself.
These tensions are amplified by a widening expertise hole: Most builders lack coaching in AI safety, and lots of safety professionals don’t totally perceive how LLMs work. In consequence, the outdated patchwork fixes really feel more and more insufficient when the fashions are writing and working code on their very own.
The rise of “vibe coding”—counting on LLM solutions with out assessment—captures this shift. It accelerates growth however introduces hidden vulnerabilities, leaving each builders and defenders struggling to handle novel dangers.
From Avoidance to Resilience
AI adoption gained’t cease. The problem is transferring from avoidance to resilience. Frameworks like Databricks’ AI Threat Framework (DASF) and the NIST AI Threat Administration Framework present sensible steering on embedding governance and safety straight into AI pipelines, serving to organizations transfer past advert hoc defenses towards systematic resilience. The aim isn’t to get rid of danger however to allow innovation whereas sustaining belief within the code AI helps produce.
Transparency and Accountability
Analysis exhibits AI-generated code is usually less complicated and extra repetitive, but additionally extra susceptible, with dangers like hardcoded credentials and path traversal exploits. With out observability instruments reminiscent of immediate logs, provenance monitoring, and audit trails, builders can’t guarantee reliability or accountability. In different phrases, AI-generated code is extra more likely to introduce high-risk safety vulnerabilities.
AI’s opacity compounds the issue: A operate could seem to “work” but conceal vulnerabilities which might be tough to hint or clarify. With out explainability and safeguards, autonomy shortly turns into a recipe for insecure methods. Instruments like MITRE ATLAS can assist by mapping adversarial ways towards AI fashions, providing defenders a structured technique to anticipate and counter threats.
Wanting Forward
Securing code within the age of AI requires greater than patching—it means breaking silos, closing talent gaps, and embedding resilience into each stage of growth. The dangers could really feel acquainted, however AI scales them dramatically. Frameworks like Databricks’ AI Threat Framework (DASF) and the NIST AI Threat Administration Framework present buildings for governance and transparency, whereas MITRE ATLAS maps adversarial ways and real-world assault case research, giving defenders a structured technique to anticipate and mitigate threats to AI methods.
The alternatives we make now will decide whether or not AI turns into a trusted companion—or a shortcut that leaves us uncovered.
Guarantee your methods stay safe in an more and more AI-driven world
Be a part of Chloé Messdaghi and a lineup of prime safety professionals and technologists for O’Reilly’s Safety Superstream: Safe Code within the Age of AI. They’ll share sensible insights, real-world experiences, and rising traits that may assist you to code extra securely, construct and deploy safe fashions, and defend towards AI-specific threats. It’s free for O’Reilly members. Save your seat right here.
Not a member? Join a free 10-day trial to attend—and take a look at all the opposite nice sources on O’Reilly.
