Qevlar AI
Speed and volume metrics can keep your SOC dashboard green while investigations get shallower. Drawing on his experience in security operations, Rafal Kitab, Director of SecOps at Abstract, explains why quality assurance has to come before any SOC metrics program, and which metrics are worth tracking once it’s in place.
“The only security metric I care about is that an organization can achieve its objectives without cybersecurity related interruptions.”
A wise man, once
My first job in cybersecurity was a junior analyst in a massive SOC with 80 clients and about 7 analysts per shift to handle them all. I still remember how one of the laziest people on the team bulk-closed a couple hundred phishing cases and the next day was noticed by our non-technical SOC manager. To us this was just a routine cleanup, for the manager it was a massive productivity increase. That analyst was rewarded for their work, and I don’t remember if it was a gift card or a smart watch, but I do remember how our collective morale tanked.
To us this felt like our hard work was not being recognized, but in fact it was just one of countless examples of how metrics used to measure security operations are not reflecting our day to day reality, performance or effectiveness.
If you’ve worked in SecOps you probably recognize this as a common theme, and if you’re leading that function right now, I want to use this opinion piece to do two things. First, look at how SOCs typically measure themselves and why those metrics say so little about how well we detect and respond. Then, explain how I think about a SOC metrics program that actually does.
I remember when I first heard that a Security Operations function should be “business aligned”. “Well, yeah,” I thought to myself, “the business doesn’t want to have security incidents, I don’t want to respond to them, we are 100% aligned.”
Turns out, this is not what it meant, but I find myself coming back to this initial understanding I had when I was just starting and can’t help but think that I was very close to being right. Preventing all incidents is the best case scenario, reducing their impact by introducing strong preventative controls and quick response is the more realistic goal.
In fact, we could define our jobs in Security Operations with even fewer words: we’re here to detect and respond to threats. Does the way we’re measuring our SOCs help us in meeting those objectives? Let’s think about it.
I like to say that what you measure is what you optimize for.
If we look at what MSSPs put in their Service Level Agreements or what in-house SOCs measure internally, we will see that it is mostly speed and volume. Mean Time To Detect, Investigate, Acknowledge, Respond / Resolve or Alert volume divided by severity or technique are some of the most frequently used and quoted SOC metrics. I’m saying this based on my work experience in SecOps from a junior analyst to a director, but also industry surveys, including one we’ve run with Ignacio Sbambato and Filip Stojkovski recently.
Now, if we measure how quickly alerts are being addressed and resolved, then that means we optimize for speed. But doesn’t that encourage surface level investigations? On one hand, we say we want quality investigations. On the other hand, we judge analysts by how fast they handle alerts or how many they can fit into their shift. Not only that, the higher the severity of an alert, the less time analysts are given to handle it.
Here is an example SLA table:
The higher the severity of an alert, the higher the potential impact it can have on an organization. Weirdly enough, the higher the stakes, the less time our analysts have to investigate. From that perspective, MTTR demands simply make no sense.
And obviously, I get why we structure our metrics this way. We want serious cases to be addressed quickly. Some SOCs realized that it makes sense to pick up alerts quickly while not rushing analysts, so they talk about MTT-Acknowledge and MTT-Respond as separate metrics. That makes sense on paper, but what’s challenging is measuring MTTA.
Our security tools generate a timestamp for alerts being assigned to analysts, which is often how MTTA is calculated (time of assignment to an analyst minus time of alert generation). But these days with auto-assignments and AI increasingly being used for triage and investigations, the technical possibilities of measuring MTTA are very limited and often simply not there.
The marketing of “machine speed attacks require machine speed response”, while not entirely wrong, feeds into that spiral of our industry’s obsession with resolving alerts in record speeds. And have you ever thought about what that does to a SOC?
Analysts given strict alert resolution targets (both speed and volume wise) look for the first sign an alert is a false positive and close it as such. They’re essentially trained to triage instead of investigate, and that muscle memory stays with them as they move up into higher SOC tiers. Even when workload or SLA requirements relax later, that way of working stays with analysts. I’ve seen many SOCs deal with this exact problem and it’s a lose-lose situation for everybody. Obviously, analysts get burned out, but the business also suffers because it’s paying for senior resources that are supposed to investigate in depth and surface threats better, and is left with essentially better paid T1 SOC analysts who keep triaging due to not having space to investigate.
Focusing on speed too much makes your SOC less effective at detecting threats long term and essentially guarantees that things will be missed. But hey, the dashboard shows green.
What always seemed odd to me is that we measure our SOCs and draw conclusions from this measurement long before we ever think of establishing a formal quality assurance program. Surely, that is completely backwards? What good is MTTA or MTTR in green if alerts aren’t resolved properly, or what does the false positive rate matter if alerts are categorized wrong?
Based on experience, third party and our own surveys, we can tell with certainty that Quality Assurance is either not systematically done, or is done only after a major threat is missed. I remember many times over my career that all of a sudden our SOC would have this push to validate, do QA, sometimes even with senior analysts specifically dedicating their time towards Quality Assurance. Later we would learn that a true positive alert was missed so our leadership thought to take action. Those initiatives have a time to live (TTL) of about two months, and then we were back to our usual, QA-less operations.
That is, unfortunately, the norm. It means that the metrics we are collecting are often misleading and we can’t really draw definitive conclusions from them.
Another interesting point is that with AI increasingly being used for triage and investigations, either as a standalone capability or baked into security stacks, there is this notion of SOC analysts moving from investigations to more of an oversight / validation position. That means doing QA after an AI agent finishes investigating, and doing so at a scale way bigger than it is done currently. With Quality Assurance not being commonplace in Security Operations I can’t help but wonder whether that transition will be as smooth as we are led to believe.
The strict SLA requirements combined with SOCs often not doing Quality Assurance lead to a very predictable outcome: SOC metrics are being gamed left and right.
This is especially visible in MSSP-client relationships, where MSSPs produce end of month SLA reports with gamed numbers and clients don’t validate them, sometimes just glancing if everything is in green as it should be. This is a perfect example of security theatre.
Over my career I was responsible for producing and validating a metric ton (hah) of such SLA reports and very often numbers were misrepresented so that the MSSP appeared compliant. Not saying this is only the MSSPs’ fault. Those practices are in place simply because the SLA demands are often unrealistic.
Here are just some examples of common tactics SOCs use to game their metrics and appear compliant.
This is more common than many would think, and yet, due to SOCs often not doing QA and SLA requirements being tight, it’s no wonder that numbers are being gamed.
Measuring often means prioritizing. If we measure speed and volume, we optimize our SOCs for handling massive workloads under tight time constraints, and are moving them away from a place where they can comfortably deliver quality, in-depth investigations.
In my opinion, thoughtless reliance on SOC metrics simply because “this is how it’s always been” causes more harm than good, and I honestly think it’s far better not to measure anything than to measure and optimize for the wrong things.
Is that a realistic approach, though? When your leadership team meets and you’re discussing your function, can you defend your program or ask for budget allocation without metrics? Or if you are an MSSP, would your client understand if you didn’t send them a monthly report?
Probably not. There are better ways to go about SOC metrics, but importantly those require a certain level of maturity and resources to dedicate to that effort. If you have both those things, here are my ideas of how we can measure our SOCs the right way.
Without a question the first thing you must do to lay the foundations of your SOC metric program is to establish a strong Quality Assurance process. This means that once a week, bi-weekly, or monthly, one of your best analysts samples alerts handled by the team to verify they were done correctly.
Correctly here means a couple of things:
As far as specific approaches, I know SOCs who have found good success in their Quality Assurance efforts by following ISO 2859-1:2026. It lays out a framework for QA and it’s solid.
Or, here’s what you can do instead:
As already discussed, metrics are meaningless if we can’t be sure that alerts were investigated correctly. And having strong QA right now means that when AI investigations become widespread, your team will be ready to step into that new AI-SOC era.
Time and volume based metrics don’t measure how well SOCs detect and respond to threats, only how fast and at what scale. There are, however, widely used metrics that, once QA is established, can shine a light on areas where a SOC does well and where it needs help. Below are some examples of what I think are good metrics to track and how they relate to your SOC’s performance.
Understanding how many investigations ended up in a security incident (a common result of a TP) is always valuable, and these rates give you a good idea of the quality of your detections.
The only caveat is that celebrating high true positive rates without addressing the underlying causes is not productive. After all, yes, you are detecting legitimate threats, but those threats made it through your defenses and that needs to be looked at. TP rate is one of those metrics where, when it’s high, the SOC is winning but the business is losing.
TP and FP rates also can’t really be compared between SOCs, because of the existence of Benign Positives.
If you’ve worked in a SOC, you’ve probably seen some version of this sad table that explains how alerts are categorized:
Actual attack? True Positive. Alert fired for some random activity? False Positive. See how easy that is?
Or rather was, because vendors started adding their own alert resolutions, like Benign Positive. And sure, I understand why. Sometimes an alert fires for an activity that looks malicious, but upon investigation ends up being benign. We probably needed a way to separate alerts that require a real investigation but aren’t malicious from clear False Positives you can close after a quick look around. The problem is that it ends up being confusing.
Why, you ask?
Consider two SOCs. SOC 1 has a SIEM that uses a Benign Positive alert category. SOC 2 operates only on True Positive and False Positive statuses. Both SOCs received 100 alerts. 50 were clear False Positives, 50 looked malicious and required further investigation, but in the end only 5 were malicious and ended up being raised as security incidents. Here’s how their numbers look:
Both SOCs did exactly the same work, and yet SOC 2 looks like it’s drowning in false positives. That’s why I’d only compare these rates within your own SOC over time, and keep BP as its own category. BP rate is actually a good detection engineering metric, meaning detection good, behavior benign.
SIEM TCO is a big number. Often the single biggest item in your security budget or close to it. It’s difficult to discuss without a ton of context, and I’ve found senior leaders to often be either unwilling or unable to grasp it.
And yes, we could do the usual “money go up, risk go down”. But what if instead we broke the big number down into individual line items?
There are many ways of doing that. The simplest one is a flat $ per data source versus how many detections you have for it. Then add context about whether those detections have ever fired and, if you’re feeling fancy, their TP / FP rates.
That’s a lot of words, so here’s a scenario. Your SIEM budget is $20,000 per month. You’re collecting $5,000 worth of firewall logs, you have 5 detection rules that use those logs, and you generated 2 firewall related alerts this month, both false positives. Firewalls are not your critical assets. This means firewall logs are 25% of your total SIEM cost, and you’re paying $1,000 per detection rule and $2,500 per alert.
Is this cost effective? I’d argue it is not.
Obviously, define what cost effective means for you and track against that. I once assumed $1,000 per detection rule and $500 per alert was the limit, went over all our detection rules and had some interesting findings. I know not all rules are meant to fire regularly, but you get the idea.
And if you can add your own risk context to all of that and say, for example, that to write detections around your crown jewel with an ALE of $X you need $Y? That’s a very mature way of communicating your budget needs.
There’s a well established practice in our industry of preparing SOC workload reports with inflated numbers and then using them as a marketing prop. I’ve ridiculed it enough times already, so I’ll focus on what alert volume can actually tell you.
Alert volume obviously matters only after fine-tuning and automation. Personally, I like to calculate it per analyst because I know roughly how many alerts an analyst can handle in a day (about 25 for an in-house analyst, about 50 at an MSSP). But what does it tell you if volume goes up one month and down the next?
Is it good if it goes down? Did we fine-tune our detections, or lose visibility somewhere? And if it goes up? Did we cover a detection gap, or did a random rule just fire a lot?
Volume itself is not an amazing metric, but what if we measured it in relation to quality? Not only does it make for a more complete metric, I believe it’s also one of the better ways to understand your SOC’s workload ceiling.
Let me explain.
There’s a breaking point where your analysts will start missing alerts entirely due to high workload. But before that point, there’s a whole range of alert volume that won’t cause analysts to miss alerts, but will impact investigation quality. That’s the danger zone you want to be aware of and avoid.
In other words, workload goes up and quality stays intact? That’s fine. Workload keeps going up and quality starts going down? That’s where you probably want to stop.
I use “% of alerts with quality issues” as shorthand for a QA number. For some SOCs it will be % of critical defects, defect rate, alerts that passed QA and so on. Also, quality issues don’t come from workload alone. Insufficient training is a common cause too, and it’s up to SOC leads to figure out which one they’re dealing with.
There are many ways to figure out which threats should get detection priority. You can look at the most likely threats, the highest ALE risks in your risk register, or your crown jewel systems, and make sure those are well covered. A mature approach probably does all three to some extent.
Once you know which systems and threats matter most to your organization, it makes sense to track your detection coverage against them. I'm not a fan of detection coverage as a general metric because it rewards breadth over depth (much like presenting your software to industry analysts). Narrowed down to specific scenarios, though, it becomes way more useful.
There are two caveats.
First, it's often difficult to account for the native detection capabilities of your existing security stack without dedicating significant time to testing. Second, with adversaries using AI, new types of threats seem to show up every day, and traditional monitoring can't always keep up with every potential detection scenario.
That second caveat gets even harder when the threat is a known vulnerability you can't patch yet, and you need to know whether someone is already exploiting it.
“That’s where a vulnerability stops being only a patching problem and becomes a SOC problem. Most of the time the SOC has no detection for that specific vulnerability, so there’s a hole in your visibility exactly where attackers know you’re exposed. AI-driven scanning means the backlog of open vulnerabilities is only going to grow, and exploitation windows are shrinking to hours. We think the answer is to watch every open vulnerability continuously for any trace of it being used, and to bring vulnerability teams and the SOC much closer than they are today. And the point Rafal makes about QA applies here too: whatever does that monitoring, you need to be able to check its work.”
Ahmed Achchak, CEO and co-founder, Qevlar AI
Those were the more “interesting” metrics, not the entirety of what I consider worth tracking. Here’s a simple summary:
Remember the analyst who bulk-closed a couple hundred phishing cases? By the metrics our SOC was judged on, they had a great day. Those numbers looked fine on paper, but they wouldn't have survived even a little scrutiny. That's how I see most SOC metrics today. We count what's easy to count, or what we've always counted, without stopping to ask whether measuring our SOCs that way actually serves us.
Some people say security metrics don't matter at all, and that what matters is the business being able to keep achieving its objectives without cybersecurity related interruptions. I believe metrics can be useful, but you need to be selective about what you track.
And most importantly – if one thing from this article stays with you is that, SOC metrics are meaningless without a solid Quality Assurance process in place. If you're going to start measuring your SOC, it makes perfect sense to start with QA.
