The 2% Result: What a 19,500-Employee Study Says About Embedded Phishing Training

A large randomized test of phishing training in a real workplace found that it barely works as commonly deployed. Researchers at UC San Diego, the University of Chicago and UC San Diego Health ran an eight-month randomized controlled experiment covering over 19,500 employees. Recent completion of annual awareness training had no significant relationship with failing a phishing simulation. Embedded "you clicked, here's a lesson" training produced a statistically significant reduction in failure rates, but of only about 2%. If your program runs on those two modes, the study says you are buying very little risk reduction.

What the study did

The paper is Understanding the Efficacy of Phishing Training in Practice by Ho, Mirian, Luo, Tong, Lee, Liu, Longhurst, Dameff, Savage and Voelker. Its design is what makes it hard to dismiss:

• Ten simulated phishing campaigns over eight months
• Over 19,500 employees at a large healthcare organization
• Randomized assignment to training conditions and to lures
• An explicit control group that received no embedded training
• Measurement of time on the training page and whether users completed it

Most program reporting has none of this. It compares one quarter's click rate to the last quarter's, with different lures and no control group.

The findings that matter for program owners

Annual training showed no measurable effect. The authors found no significant relationship between how recently someone completed annual awareness training and whether they failed a simulation.

Embedded training helped, barely. Across all phishing emails, the average difference in failure rate between the control group and the training groups was 1.7%.

Lure choice swamped training. Some lures drew clicks from 1–2% of users. Others drew failure rates upwards of 30%. The difficulty of the email moved results far more than training did. This is the same point the NIST Phish Scale makes, covered in Why Click Rates Lie.

Most people eventually click. Over 56% of users clicked a phishing link at some point in the study, trained or not. In any single simulation the median failure rate was 10%, which means most users received no embedded training in a given round.

People do not read the lesson. Over half of embedded training sessions ended within 10 seconds. Between 37% and 51% of sessions showed no engagement at all: the page was closed immediately. Only 24% of training sessions were formally completed.

More static training did not mean fewer clicks. Users who completed multiple static training sessions had a 15% higher chance of failing a future simulation for each additional session completed. Interactive training taken to completion was linked to a 19% lower likelihood of failing. The authors note the absolute effect was small, few users completed it, and the result may reflect selection: the people who finish interactive training may differ from those who don't.

The difficulty of the lure moved results far more than the training did.

The failure mode: the completion illusion

Call it the completion illusion. The program reports completion percentages for annual training and click rates for simulations. Both numbers look like evidence of control. Neither measures whether behavior changed. Completion shows that someone reached the end of a module. A click rate without a control group and difficulty rating mostly reflects how hard the lure was.

The illusion persists because both numbers are easy to collect and easy to improve. Make the module shorter and completion rises. Make the lure easier and clicks fall. Nothing about exposure has changed.

What to change

The study does not say awareness work is pointless. It says the common forms of it don't deliver much, and that phishing defense should lean harder on controls that don't depend on people noticing. The authors point to technical countermeasures such as hardware multi-factor authentication and password managers that fill credentials only on the correct domain.

Ordered steps for awareness and security leads:

1. Stop reporting raw click rate as an outcome. Report it only alongside lure difficulty, and against a control or baseline cohort where you can.
2. Run a holdout group. Exclude a small randomized group from embedded training for a period. Without one, you cannot tell whether training changed anything.
3. Measure engagement, not completion. Track time on page and how many sessions are closed immediately. If most sessions end in seconds, the lesson isn't being delivered.
4. Retire repeat static modules. If the same static page is shown again and again, the study's data gives no reason to expect it to help, and some reason to expect it won't.
5. Move effort toward interactive formats. They were the only modality in the study linked to a significant improvement, with the selection caveat above. Pilot one and measure it against a holdout.
6. Shift the phishing budget toward phishing-resistant authentication. Training cannot fix a credential that is valid wherever it is typed. A factor bound to the real domain can. The same reasoning applies to SMS OTP in banking, covered in Beyond SMS OTP.
7. Keep the reporting channel and measure it separately. The study did not examine reporting, so it neither supports nor undermines it. Track report volume and time-to-report on their own terms.

What to tell the board

Boards that have been shown falling click rates as proof of a working program deserve a correction. Keep it short:

• The organization's own trend line may reflect lure difficulty, not behavior.
• A large randomized study found annual training had no measurable effect and embedded training had a small one.
• The program is being restructured to measure real engagement against a control group.
• Investment is moving toward authentication controls that work even when someone clicks.

That message is uncomfortable. It is still better than defending a metric that a peer-reviewed experiment has already undercut.

Limits worth stating

The data comes from one healthcare organization. The authors say results may not generalize to every sector. They also note their findings align with other randomized studies in different organizations and sectors. Click-through was the measure of failure; credential entry and attachment execution were not studied. Treat the 2% figure as the effect of training as commonly deployed, not as a ceiling on what better-designed training could achieve.