What You Measure is the Program

Last month's close was a claim about the program. Hygiene is the floor, context is the ceiling, and the gap between them is the program. If that gap is the program, the next question is the one every board eventually asks. How do you measure it.

The industry has an answer, and it is the wrong one. We measure activity. Alerts processed. Vulnerabilities counted. Actions taken. None of those numbers tell you whether risk went down. June was the month I put that argument on the record twice, in two different rooms, and the responses confirmed what the April events already implied. The measurement model is broken, and the people closest to the work know it.

The activity trap. Start with the Security Operations Center (SOC). Walk into any of them and you find the same two things. An automation platform, and a backlog larger than last quarter. The relationship between those two facts is the part nobody wants to say out loud.

Alert-only automation does not reduce work. It re-categorizes it. The hours that used to go into investigation now go into response, and the response is still manual. The platform produces a higher-fidelity ticket, and a human still takes the action. We got faster at generating tickets we cannot close.

The Security Orchestration, Automation, and Response (SOAR) generation is the cleanest proof. The promise was that playbooks would handle the routine eighty percent so analysts could work the twenty percent that mattered. In practice the playbook catalog became a maintenance burden. Every runbook ages with the environment, and teams find out their automation stopped firing only when something breaks at 3 a.m. That is not automation. That is deferred work with a dashboard on top.

The fix is not a better alert. It is execution. Humans should approve actions, not decide which alerts get a runbook attached.

The counting trap. The vulnerability program has the same disease in a different organ. We count. Open vulnerabilities, closed vulnerabilities, mean time to patch. A board reads those numbers and learns nothing about whether the organization is exposed.

Last month's NVD retreat made the count emptier. When the National Vulnerability Database (NVD) stopped enriching most Common Vulnerabilities and Exposures (CVEs), it took the centralized severity context with it. The number you used to lean on now has to be re-derived inside your environment, against your assets, with your exploitability signal. A raw count was always weak. Without enrichment behind it, it is noise.

Counting vulnerabilities measures how much you found. Proving risk measures whether you are safer. Those are different questions, and only one of them survives contact with an audit committee. The shift is from "how many" to "prove it went down."

The pattern

Read the two together and it is one mistake wearing two uniforms. Activity metrics reward motion. Outcome metrics reward results. Alerts processed and vulnerabilities counted are activity. Backlog reduced and risk proven down are outcomes. The first set is easy to produce and impossible to defend. The second set is harder to produce and reads in a boardroom without translation.

This is the context-is-the-ceiling thesis made operational. Context is what converts a pile of activity into a defensible outcome. It tells you which dormant account is safe to disable, which CVE is reachable in your environment, and which action moved the risk register. Without it, you are measuring effort. With it, you are measuring effect.

Here is the test for any program. If your strongest metric is a count of things you did, you are measuring activity. If your strongest metric is a reduction the board can verify, you are measuring outcomes. Dormant accounts older than ninety days. Expired certificates renewed before expiration. External authorization grants reviewed within thirty days. Privileged service accounts with no business owner of record. Those numbers move when execution lands, and they stay flat when an organization keeps buying faster triage.

"Activity metrics reward motion. Outcome metrics reward results."

From National Harbor

I took this to the Gartner Security and Risk Management Summit at National Harbor this month, and the floor conversations carried the through-line from RSA and from April. Operators are tired of activity dashboards. The question underneath nearly every exchange was the board's question, asked plainly. Can you prove the risk went down. The people with an answer to that are getting follow-ups. The ones selling faster triage are getting polite nods.

Don't take my word for it

I made both arguments in full this month. Read them, push on them, and tell me where they break in your environment.

• Against Alert-Only Automation — Surf AI Blog
• Stop Counting Vulnerabilities, Start Proving Risk — Forbes Tech Council

That gap between activity and outcome is exactly what we built Surf to close. April did not change the roadmap. June sharpened the argument for it.

See you at Black Hat

I'll be at Black Hat USA in Las Vegas, August 4 through 6. If you're heading out, come find us at booth 4711. We're running the Surf Spa this year, so you can book a 10-minute massage and actually sit down for a few minutes in the middle of a long week. Time it right and you'll catch me playing guitar too. Bring the board's question with you and we'll get into it in person.

See what we have planned and grab a slot.

More soon.

Yonesy Núñez
CISO, Surf AI