Most security teams report a single number to leadership every quarter: the phishing simulation click rate. It goes down, everyone relaxes. It goes up, someone gets a stern email about "the human firewall." Neither reaction is justified, because the number being tracked doesn't measure what people think it measures.
A 2025 study by Andrew Rozema and James Davis covering 12,511 employees at a US financial technology firm confirmed something practitioners have suspected for years: click rate is far more sensitive to lure difficulty than to training. Training interventions showed no significant effect on click or reporting rates. Lure difficulty, scored with the NIST Phish Scale, did: click rates rose from 7% on easy lures to 15% on hard ones — more than double, from the same population. If your vendor quietly makes next quarter's simulation easier, your click rate improves. Nothing about your organization's actual susceptibility changed.
The metric was never broken. The measurement discipline was.
The NIST Phish Scale exists precisely to solve this. It scores every simulated lure along two independent axes:
- Observable cues — typos, spoofed domains, generic greetings, urgency language. The more cues present, the easier the lure.
- Premise alignment — how closely the pretext matches an employee's actual daily workflow. An invoice-approval lure aimed at finance staff has far higher alignment than a generic "IT password reset" sent to everyone.
A click rate reported without its corresponding Phish Scale score is not a metric. It's an anecdote wearing a percentage sign.
The practical failure mode: organizations that don't baseline lure difficulty end up in one of two traps. Either they send easy lures indefinitely and mistake a flat 2–3% click rate for a mature security culture, or they escalate difficulty without warning and blame a spike on "declining vigilance" when it was actually a harder test.
What operationalizing the Phish Scale actually looks like
- Score every simulation before it ships. Assign an observable-cues rating and a premise-alignment rating using the NIST scale's published criteria — not a vendor's proprietary "difficulty" label, which is rarely transparent about methodology.
- Report click rate paired with its Phish Scale band, not as a standalone trendline. A board slide showing "6% click rate, high-alignment lure" is a meaningfully different claim than "6% click rate, low-alignment lure."
- Hold difficulty constant across comparison periods. If you want to measure quarter-over-quarter improvement, the lure difficulty band has to be held roughly steady, or you're comparing two different experiments and calling it one trend.
- Stop optimizing for a falling click rate as the end goal. A program that only ever sends easy lures will produce a beautiful, meaningless graph. Difficulty should trend up as your workforce matures, with click rate tracked relative to that shift — not in isolation.
Why this matters more than most security leaders assume
A second large study found something more uncomfortable. In an eight-month randomized experiment with more than 19,500 employees at UC San Diego Health, Liu, Ho, Mirian and colleagues (IEEE Symposium on Security and Privacy, 2025) found that users who completed multiple static embedded training sessions had an 18.5% higher likelihood of failing for each additional session completed. More training of that kind correlated with worse outcomes, not better ones. A click-rate-only dashboard cannot detect this. A Phish Scale-aware program can, because it separates "the test got easier" from "the workforce got better" — and those are opposite conclusions that a raw percentage cannot distinguish.
If your board is currently being shown a single click-rate number without a difficulty baseline attached to it, that number is not evidence of program effectiveness. It's evidence of whatever your vendor's template shipped with this quarter.
If your organization is still reporting click rate as a standalone number, this is the gap our Ethical Hacking & Bug Bounty Bootcamp's human-risk module was built to close — teaching teams to design and score simulations the way NIST does, not the way a vendor dashboard defaults to.