Picture the surveillance audit. The awareness program is presented with justifiable pride: 100% completion of the annual security training, quarterly phishing simulations, a click rate falling steadily for three years. The auditor notes conformity with Clause 7.3 and moves on. Everyone in the room believes the organization has become measurably harder to phish.
In 2025, that belief was finally tested with the rigor ISO management systems are supposed to embody, and it did not survive.
What the Evidence Actually Shows
The pivotal study was a randomized controlled trial published at IEEE Security and Privacy 2025, covering 19,789 employees of UC San Diego Health over eight months. Its design matters for anyone who thinks in terms of audit evidence: random assignment, control groups, and real behavioral outcomes rather than self-reported confidence.
The findings, in ascending order of discomfort. Annual awareness training showed no significant relationship between training recency and simulation failure; employees trained weeks earlier failed at the same rate as those a year overdue. Embedded post-click training produced an absolute improvement of under two percentage points, explained largely by engagement data showing a third to a half of employees closed the training page in zero seconds. And employees who completed multiple rounds of static embedded training exhibited 18.5% higher failure odds per additional session. The intervention deployed to reduce the risk was, for a substantial population, associated with increasing it.
This corroborates rather than contradicts the earlier literature. An ETH Zurich field study of 14,733 employees over 15 months found the trained group subsequently clicked more than the untrained control, and identified a plausible mechanism: 43% of employees reported feeling safer after completing the training. The training delivered a sense of protection instead of protection, which is the definition of a control that has become a liability. A 2025 replication at a US financial services firm, with lure difficulty controlled using the NIST Phish Scale, found no significant training effect on either clicking or reporting.
For compliance professionals, the correct reading is precise: the evidence does not say awareness is worthless. It says the dominant delivery model, completion-tracked modules plus simulation click rates, produces activity without measurable effectiveness, and sometimes negative effectiveness.
This Is a Clause 9.1 Problem Wearing a Clause 7.3 Badge
ISO/IEC 27001 never asked for phishing simulations. Clause 7.3 requires that persons be aware of the information security policy, their contribution to the effectiveness of the ISMS, and the implications of nonconformity. Annex A control 6.3 requires awareness, education and training that is relevant to the person’s role, with the supporting guidance in ISO/IEC 27002 emphasizing that awareness activity should be evaluated for effectiveness. And Clause 9.1 requires the organization to determine what needs monitoring, and to evaluate whether the ISMS achieves its intended results.
Now hold the standard program against those requirements. A training completion rate is a record that an activity occurred, which satisfies the documentation habit but says nothing about awareness outcomes; the zero-second engagement data shows exactly how wide that gap can be. A falling click rate looks like an effectiveness measure but fails basic measurement validity: in the UCSD trial, failure rates across lure templates ranged from roughly 2% to over 30%, meaning the metric responds more strongly to template selection than to workforce capability. A program can show three years of improving click rates while its people become no harder to deceive, and vendor before-and-after curves, constructed without control groups, should be treated with the same skepticism an auditor would apply to any supplier’s uncontrolled performance claim.
An internal auditor who accepts completion percentages and click rates as effectiveness evidence is, in ISO terms, accepting nonconforming measurement. The nonconformity just happens to be popular.
What Effective Looks Like, in the Language of the Standard
The same research literature that dismantles the standard model points clearly at what to build instead, and each element maps onto the ISMS rather than around it.
Measure reporting, not clicking. The ETH study found employee reporting operationally viable as a detection channel, with 68% accuracy and a manual triage load of roughly 1.5 emails per day, and the 2025 replication found that in over a third of campaigns someone reported the phish before anyone clicked. Reporting rate and time-to-first-report are genuine effectiveness metrics for Clause 9.1, and they connect awareness directly to the incident management controls (A.5.24 through A.5.26) where its value is realized. A workforce that reports fast is a functioning detective control; a workforce with a low click rate on easy templates is a statistic.
Prefer controls that do not require vigilance. The UCSD authors’ recommendation was not better training but phishing-resistant authentication and password managers. That is Annex A 8.5 territory, and the current threat environment reinforces it: adversary-in-the-middle kits now bypass one-time-code MFA at industrial scale, while FIDO-based passkeys remove the phishable credential entirely, with large-scale deployment data showing usability gains alongside the security ones. Risk treatment that shifts load from human judgment to technical control is simply proportionate treatment based on the evidence available.
Keep awareness, change its form. The follow-up ETH research published at ACM CCS 2024 found that brief periodic reminders performed as well as full training content, and that repeat clickers were failing through inattention rather than ignorance, which no module fixes. Short, frequent, blameless nudges satisfy the intent of Clause 7.3 better than the annual module, at a fraction of the cost, and without teaching employees that security is a trick played on them by their own organization. The leadership dimension of awareness culture matters more here than any content library: people report quickly when reporting is safe.
Retain simulations only as measurement. Used sparingly, unannounced to no one’s punishment, and analyzed with difficulty held constant, simulations can measure whether reporting works and whether technical controls catch what humans miss. Used as training theatre, the evidence says they produce false confidence and audit-friendly numbers.
The Question for Your Next Management Review
The uncomfortable summary is that a widely deployed control, purchased in good faith and documented impeccably, has failed its effectiveness evaluation in the largest trials ever run, and most ISMS documentation has not yet noticed. There is a certain irony available to organizations that pride themselves on investing in security awareness: the investment is sound; the instrument is not.
The fix begins with one question in the next management review, asked exactly as Clause 9.1 intends: what evidence do we hold that our awareness program changes behavior, and would that evidence survive the scrutiny we apply to every other control? If the honest answer is a completion rate and a click-rate chart, the program has been auditing its own activity, and the organization deserves better data than that.
Awareness is still a requirement. Effectiveness always was.
