Cybersecurity threats in the age of generative AI

The uncomfortable fact is this: phishing remains the entry point for more than 80% of cyberattacks, and generative AI is not replacing the old criminal playbook so much as compressing it. Reconnaissance, impersonation, vulnerability scanning, credential abuse — the dull labor of intrusion — can now run faster, cheaper, and with less visible incompetence. In some cases, AI-driven attacks can cut reconnaissance and vulnerability scanning time by 50% to 70%. That is not science fiction. That is margin expansion for criminals.

And yes, I used the phrase margin expansion deliberately. Cybercrime is a business. A grubby one, but a business. When the cost of customer acquisition falls — here, the “customer” is your accounts payable team, your VPN portal, your neglected legacy application — volume rises. The market clears at more fraud.

The phishing email no longer looks like it was translated through a toaster

For years, security awareness training leaned on a comforting fiction: people could spot phishing because phishing looked ridiculous. Bad grammar. Odd salutations. A fake invoice from a vendor you had never used. A domain name with one letter swapped by someone who apparently learned English from airport signage.

That era is dying, and good riddance to the lazy training that depended on it.

Generative AI tools now produce clean, plausible, role-specific emails at scale. They can mimic tone, strip out spelling errors, adapt to industries, and tailor messages to the rhythm of corporate life: quarter-end close, procurement approvals, performance reviews, legal escalations, software renewals. The email no longer says, “Dear esteemed beneficiary.” It says, “Can you confirm whether the revised SOW for the cloud migration was routed through Coupa or DocuSign? Legal wants this cleared before 4 p.m.”

That is a different animal.

Let me translate this for you: the attack has moved from obvious deception to operational friction. It does not need to hypnotize anyone. It only needs to arrive at the exact point where a busy employee is already annoyed, overloaded, and trying to clear a queue before a meeting.

The best AI-generated phishing messages tend to exploit three corporate weaknesses:

1. Process ambiguity. If nobody knows whether finance approvals happen in email, Slack, Teams, an ERP workflow, or a spreadsheet owned by a person named Denise, the attacker does not need genius. They need timing.

2. Executive impatience. A message that appears to come from a senior leader demanding “urgent discretion” still works because organizations keep rewarding speed over verification.

3. Vendor sprawl. Every enterprise now has a small nation-state’s worth of SaaS providers. Attackers can imitate renewal notices, invoice disputes, support escalations, and access requests with depressing ease.

The old advice — hover over links, look for typos, trust your gut — now feels like handing out umbrellas in a hurricane. Useful, perhaps. Not sufficient.

AI did not invent phishing. It removed the amateur smell from it.

This matters because cyber defense budgets often still treat phishing as a people problem rather than a system design problem. Train the staff. Blame the intern. Send another cartoonish simulation. Then act surprised when the breach report reads like a Greek tragedy written in procurement language.

A serious posture assumes humans will be pressured, distracted, and socially engineered. Controls must absorb that reality. Payment changes need out-of-band verification. Privileged requests need approval paths that cannot be rewritten by a persuasive email. Vendor communication needs known channels, not improvisational theater.

Security, at its best, is not a lecture. It is architecture.

Large language models have industrialized the boring parts of intrusion

The public conversation around generative AI security risks gets distracted by cinematic nonsense: rogue superintelligence, glowing red terminals, a hoodie-clad villain asking a chatbot to “hack the Pentagon.” Please. Most enterprise damage comes from something more prosaic: repetition at scale.

Large language models help criminals draft lures, write scripts, summarize stolen documentation, translate technical notes, and accelerate vulnerability research. They do not need to be magical. They only need to remove friction.

One useful way to think about modern malware trends is that AI has widened the middle class of cybercrime. Not every attacker becomes elite. But many become competent enough to be dangerous.

A novice who once bought a crude phishing kit can now generate better copy, test variations, and localize attacks across languages. A mid-tier operator can use LLMs to review leaked code, identify likely weak points, and automate reconnaissance against legacy systems. A more capable group can fold AI into vulnerability discovery pipelines, sifting exposed services, misconfigured cloud assets, and stale software dependencies faster than a human analyst working alone.

That is where the leverage sits.

The enterprise risk is not that every criminal suddenly becomes a genius. The risk is that ordinary criminals become productive.

Now, before the hype merchants start panting into their microphones, let me be precise. AI is an accelerator, not the sole cause of rising cyberattacks. Criminal ecosystems were already mature. Ransomware groups already had affiliates, help desks, negotiation scripts, and revenue-sharing models. Cloud misconfigurations existed before anyone asked a chatbot to write Python.

But acceleration changes markets. I watched this happen in 2008 when leverage turned bad assumptions into global contagion. The instrument did not create human greed; it multiplied the blast radius. Generative AI is playing a similar role in cybercrime. It scales intent.

The most exposed companies are not always the ones with no security team. They are often the ones with too many systems, too many exceptions, too much technical debt, and a leadership team that mistakes security tooling for security discipline. A dashboard is not a defense strategy. It is a screen with colors.

Credential stuffing has learned to adapt

Credential stuffing used to be brutish: take stolen username-password pairs, throw them at login portals, hope enough employees reused passwords from some long-dead consumer breach. Crude, yes. Effective, also yes, because human password behavior remains a museum of poor choices.

AI makes this uglier by improving adaptation. Automated credential stuffing attacks can analyze responses, change patterns, route attempts, and adjust to security challenges in something closer to real time. The attack becomes less like a battering ram and more like a swarm of pickpockets testing every pocket on the train.

Enterprises often underestimate this because each failed login looks small. One account. One IP. One blocked attempt. A little noise in the SIEM. But scaled across regions, devices, identity providers, SaaS applications, and contractors, that noise becomes a liquidity problem for the security team. Too many alerts. Too many borderline cases. Too little confidence.

The criminal does not need to win everywhere. They need one reused password, one stale account, one service account with embarrassing privileges, one employee who approved a push notification because it arrived during a commute.

The defensive implications are not glamorous, which is probably why they do not get enough keynote airtime:

• Password reuse must be treated as an enterprise exposure, not a personal failing. If employees use corporate email addresses across consumer services, breach data eventually comes home to roost.
• Multi-factor authentication has to be phishing-resistant where it matters most. Push fatigue and one-time codes are better than nothing, but they are not the top shelf. Hardware-backed and passkey-based approaches reduce the attack surface for high-risk roles.
• Dormant accounts deserve suspicion. Former contractors, test users, and abandoned admin accounts are not harmless clutter. They are inventory for thieves.
• Identity telemetry needs business context. A login from a new geography is one thing. A login from a new geography followed by invoice export, mailbox rule creation, and privilege escalation is quite another.
• Rate limits and bot defenses must evolve. Static defenses invite adaptive abuse. Attackers are already iterating; defenders do not get moral credit for standing still.

The identity layer has become the new perimeter, which is one of those phrases consultants say until it loses oxygen. But in this case, the cliché points at something real. Employees authenticate into everything: finance systems, source repositories, customer data platforms, HR portals, cloud consoles. Once credentials become portable leverage, the attacker no longer needs to smash the front door. They borrow a badge.

Deepfakes turn executive authority into a spoofable asset

Business Email Compromise was already the white-collar jewel of cybercrime: fewer explosions, more wire transfers. Now AI-generated voice and video deepen the con. Fraudsters can impersonate executives, vendors, legal advisers, and investors with enough realism to push an employee across the line from “this seems odd” to “the CEO is on the call and sounds irritated.”

Deepfake-related fraud incidents have been rising in corporate environments, and the direction of travel is obvious even if exact global loss figures attributable specifically to AI remain hard to isolate. The more remote and distributed a company becomes, the more it depends on mediated trust: screens, voices, profile photos, calendar invites, email threads. Each one is now forgeable.

This is where leadership hubris becomes expensive.

Executives often demand speed, secrecy, and deference — exactly the conditions fraudsters love. The classic BEC script thrives on hierarchy: “I need this handled now,” “Do not loop in others,” “This is confidential,” “The board is waiting.” Add a convincing voice clone or video snippet and the psychological pressure spikes.

And because finance teams are trained to serve the business, not interrogate it like customs officers, the fraud works. Nobody wants to be the analyst who delayed a strategic acquisition wire because the boss sounded slightly synthetic.

Here is the sane rule: if money moves, identity must be verified through a channel the requester does not control.

Not “reply to the email.” Not “the voice sounded right.” Not “the Zoom background matched.” A known callback number. A pre-established approval workflow. Dual authorization. Segregation of duties. Boring controls. Beautifully boring controls.

The future of fraud will not always look like a hack. Sometimes it will look like leadership doing what leadership always does: demanding urgency without accountability.

The companies that handle this well will make verification culturally normal. Not insulting. Not bureaucratic. Normal. When a controller calls to confirm a wire instruction, the CFO should thank them, not perform wounded authority. The cheapest security control in finance is a senior executive with enough humility to tolerate being checked.

A rare creature, admittedly.

Model poisoning: the quiet attack on the machine’s memory

If phishing attacks people and credential stuffing attacks identity, model poisoning attacks the assumptions of AI itself.

The basic idea is simple enough: attackers manipulate training data or inputs so that an AI model learns the wrong thing, develops a hidden weakness, or behaves in a biased or exploitable way. In security-sensitive environments, that can mean backdoors, classification errors, corrupted recommendations, or blind spots in detection systems.

This is not the most visible category of emerging cyber attacks, but it may become one of the more consequential as enterprises stuff AI into workflows they barely understand. Fraud detection. Code review. Customer support. Threat triage. HR screening. Document analysis. Supply chain monitoring. Every place a model interprets data becomes a place where poisoned data can whisper instructions into the system’s ear.

The risk grows when companies treat AI models as oracle boxes rather than engineered systems with supply chains. Data comes from somewhere. Labels come from somewhere. Fine-tuning sets, plugins, retrieval databases, third-party APIs, and user feedback loops all create attack surfaces. The model is not just “the model.” It is an ecosystem with a memory, an appetite, and occasionally the judgment of a sleep-deprived analyst.

This is especially relevant for enterprise software and digital infrastructure markets outside the usual Silicon Valley theater. As founders build less glamorous but more essential systems — billing engines, workflow automation, identity layers, logistics software — they inherit security obligations from day one. The same dynamic is visible in markets betting on enterprise software and digital infrastructure as the next technology boom, where the opportunity is real precisely because the infrastructure is becoming more consequential. Consequence attracts capital. It also attracts attackers. Funny how that works.

Model poisoning is nasty because it can evade the mental model executives use for cyber risk. They understand stolen passwords. They understand ransomware screens. They sort of understand phishing, at least after legal has frightened them. But poisoning sounds abstract until a fraud model starts letting through the wrong transactions or a security classifier learns to ignore a specific payload pattern.

The defense starts with governance that has teeth:

• Know what data trains or tunes the model. If the answer is “a vendor handles that,” congratulations, you have discovered a liability, not a strategy.
• Track data provenance. Training and retrieval sources need lineage, versioning, and integrity checks.
• Test models adversarially. Do not merely ask whether the model performs well on friendly benchmarks. Ask how it fails when someone wants it to fail.
• Separate high-risk decisions from full automation. AI can triage, recommend, and accelerate. For material financial, operational, or security decisions, human review and audit trails still matter.
• Monitor drift and anomalies. A model that performed well last quarter may degrade or behave oddly after data shifts, user feedback manipulation, or hostile inputs.

Regulators have begun formalizing AI security expectations in the US and EU, and organizations such as NIST have pushed frameworks for managing AI risk. That is useful, but let us not pretend regulation will save careless firms from themselves. Compliance often arrives as theater after the architecture is already rotten.

The board question should be blunt: where are we letting AI make or influence decisions, what could an attacker gain by corrupting that process, and who owns the control environment?

If nobody can answer, the risk is not theoretical. It is merely unpriced.

Defense has to become faster without becoming stupid

The obvious response to AI-accelerated attacks is AI-driven defense. Obvious, and only partly satisfying.

Security teams do need automation. They need detection systems that correlate signals across identity, endpoint, cloud, email, and application layers faster than humans can. They need tools that summarize incidents, prioritize alerts, spot behavioral anomalies, and help analysts move from noise to action. In a world where attackers use AI to compress time, defenders cannot insist on artisanal incident response.

But the mirage is believing AI defense is foolproof. It is not. No model is unhackable. No platform removes the need for good asset inventory, patch discipline, access control, incident rehearsal, and executive judgment. Anyone selling “autonomous security” as a magic exorcism should be made to explain lateral movement to a room full of exhausted SOC analysts at 2 a.m.

The practical shift is toward resilience, not fantasy prevention. Assume some attacks will get through. Then reduce blast radius.

That means a few hard, unfashionable things:

1. Shrink privilege. Over-entitled users and service accounts are balance-sheet liabilities with login credentials. Least privilege is not elegant. It is profitable.

2. Patch the systems that attackers actually target. Legacy systems are not charming. They are deferred expenses with network access.

3. Segment critical environments. If one compromised account can wander from email to finance to cloud administration, your architecture has chosen drama.

4. Run payment fraud drills. Do not wait for a deepfake incident to discover that nobody knows who can stop a wire.

5. Instrument the identity layer. Login behavior, device trust, impossible travel, privilege changes, mailbox rules, token abuse — this is where the smoke appears before the building burns.

6. Protect the AI stack itself. Models, prompts, training data, retrieval systems, plugins, and APIs all need security review. Treat them like production infrastructure because that is what they are.

7. Make verification socially acceptable. If employees fear punishment for slowing down a suspicious request, attackers already own part of your culture.

The better security organizations are not the ones buying every shiny product. They are the ones reducing friction for legitimate work while increasing friction for illegitimate authority. That distinction matters. Bad security punishes employees for doing their jobs. Good security makes the dangerous shortcut harder than the safe path.

I have seen companies spend seven figures on tooling while leaving shared admin credentials in a spreadsheet. I have watched boards nod gravely through cyber presentations packed with red-yellow-green dashboards and ask not one question about recovery time, third-party access, or who can approve a payment exception. The aesthetic of seriousness is not the same as seriousness.

Generative AI will expose that gap.

The real risk is speed meeting complacency

Cybersecurity threats in the age of generative AI are not mysterious because the technology is incomprehensible. They are dangerous because they collide with habits businesses already had: sloppy identity management, messy vendor processes, executive impatience, underfunded modernization, and a touching belief that awareness training can compensate for broken workflows.

AI makes the attacks cleaner. Faster. More personalized. More adaptive. It gives mediocre criminals better leverage and competent criminals more scale. It turns phishing into tailored persuasion, credential stuffing into adaptive pressure, malware development into a more efficient assembly line, and model manipulation into a new class of quiet sabotage.

The answer is not panic. Panic is expensive and usually poorly procured. The answer is disciplined paranoia: verify authority, harden identity, reduce privilege, modernize the ugly systems everyone pretends not to see, test the AI stack, and build defenses that assume humans will be human.

Innovation deserves respect when it creates real value. Generative AI does. But markets do not reserve leverage for the virtuous. Criminals get the productivity gains too.

That is the part the keynote slides tend to leave out.