Qevlar AI
Moving an AI SOC from pilot to production takes more than accurate verdicts. Discover how Qevlar uses graph orchestration to make investigations reproducible, Sampling to measure agreement with analysts on live alerts, and customer-specific context to scale across MSSP environments. Learn how continuous validation enables SOC teams to move from manual review to trusted, controlled automation.
With standard access to a frontier model, an engineer can spin up an AI alert investigator within a single sprint. MSSP security teams are increasingly building these in-house, drawn by how quickly a prototype comes together and by full control over its logic. The pilot reflects the team's own context and tools from day one, and on many alerts its verdicts already look sound.
A pilot and a production system are different things, however. An AI SOC enters production on the day analysts start acting on its verdicts without redoing the investigation. Until then, every conclusion it produces is a recommendation someone still has to check.
Analysts act on a verdict only when they can reproduce it, follow the reasoning behind it, and see evidence that the system agrees with them. All three are hard to achieve with a prototype. This article covers what it takes to close that gap, and what trust makes possible once its earned.
A pilot answers one question well: can this system reach a good verdict on the alerts we gave it?
Production depends on two further questions that the pilot leaves open:
The first question carries more risk than it appears to, because an inconsistent system does not fail in one predictable direction. Run often enough, the same alert can be escalated on one occasion and closed on another. A malicious alert closed as benign on the wrong run is a missed attack, and nothing in the system marks that run as different from the ones that caught it.
The effect reaches past the individual alert. Severity drives prioritisation, so an alert rated critical on Monday and medium on Tuesday competes differently for an analyst's attention. Any response tied to that verdict fires on one run and stays idle on the next. Inconsistent conclusions also give conflicting guidance for the detection tuning that should follow an investigation.
That leaves analysts two options:
Qevlar is designed so that analysts never face that choice. The platform plans each investigation, and LLMs handle only targeted steps such as parsing a log or reading an email, so the same alert follows the same path to the same verdict. Every investigation shows each step with the observable it examined, the tools it used, and what it found, so an analyst can follow the reasoning from alert to verdict without rebuilding it.
Consistency is the property most often assumed and least often measured, so Qevlar's AI research team measured it directly. They took real production alerts that needed between 3 and 20 steps to resolve, selected 10 at each level of complexity, for 180 anonymised scenarios. An LLM agent was given a full catalogue of investigation actions, each alert was submitted 100 times, and the complete path of 18,000 runs was recorded.
Even simple alerts proved less stable than expected. Their most common investigation path rarely appeared in more than about 75% of runs. On one alert concerning suspicious outbound traffic, 17% of runs took a two-step shortcut that skipped threat intelligence enrichment altogether.
Complex alerts spread much further. An email alert that averaged 17.2 steps produced 90 distinct paths across 100 runs, and its most common path accounted for only 3% of them. Each additional step is another point at which the agent can choose differently or stop early, so variability grows with the length of the investigation.
The verdicts moved with the paths. Identical alerts sometimes received different severity ratings or threat classifications depending on the route the agent took. When every run follows its own course, analysts have no stable baseline against which to review the work, validate it, or learn from it.
That’s why Qevlar never leaves the investigation path to an LLM. Graph orchestration defines the workflow, and each step is prioritised from the alert's characteristics, threat intelligence, and historical effectiveness data. An alert that needs threat intelligence enrichment gets it on every run, because the step sits in the plan instead of depending on the model's choice in the moment.
The most direct evidence of trustworthiness is a comparison between the AI's verdicts and the analysts' verdicts on the same live alerts, made without either side seeing the other's conclusion. A share of incoming alerts has its AI verdict sealed, analysts investigate those alerts in the normal way, and the two conclusions are compared afterwards.
The sealing is what gives the measurement its value. An analyst who sees the AI's verdict first is reviewing an answer, and a review tends to confirm what it has been shown. With the verdict hidden, the analyst's conclusion is independent, and the agreement rate reflects two separate judgements of the same evidence.
Three details decide whether the result is evidence or reassurance:
For a team building in-house, this is a second system to build. Alongside the deterministic scripts that keep investigations stable, it needs a way to measure agreement with analysts, and then it has to maintain both.
Qevlar runs this measurement inside the platform with Sampling. Analysts receive a sample of Qevlar’s decisions to investigate blind, including ones it closed on its own, and the plaform tracks how often they agree. The same mechanism benchmarks Qevlar against the team's own analysts on an ongoing basis and detects drift in production, so the team always knows whether the AI still agrees with it.
.png)
For an MSSP, each of these measurements multiplies. A single organisation's build has to be consistent and trustworthy in one environment. An MSSP's build has to be consistent and trustworthy in every customer environment it serves, with an integration for each security product in each environment, maintained every time that product changes.
What an MSSP needs is stability within each customer and appropriate differences between them. That depends on context the system often lacks. Much of it lives in email threads and in the memories of the analysts who know each account, and asking analysts to type it into a tool tends to work for a week before usage drops off.
Qevlar treats each end customer as its own environment, with its own scope, context, and feedback loop, and support of custom alerts. Context is drawn from where it already lives: tickets and analyst feedback, so analysts are not asked to retype what they know.
Where context does need to be set by hand, it can be defined once for a whole range of entities, using CIDR blocks for IP ranges and wildcard patterns for other observables, instead of one observable at a time.
.gif)
The integration burden shrinks in the same way. Each investigation plan is built from the alert's own entities with no playbook per alert type, so coverage grows without a new build per use case. Connectors are vendor-agnostic, so investigations keep running through a SIEM migration or when an acquired business brings its own stack.
Models change on a schedule the security team does not set. Across OpenAI, Anthropic, and Google, 40 model versions carry a 2026 retirement date, and every model change means retesting every use case. A build that performed well on last quarter's model has to earn the same result on the next one.
Models also drift. The further attacker techniques move from a model's training data, the less reliable its judgment becomes. That decline is gradual, and without a running comparison against analysts, there is nothing to show it happening.
Qevlar runs on the model the customer chooses, in their cloud or on-premises, and absorbs what changes around it. Because graph orchestration limits the model to targeted steps, a model change touches those steps rather than the whole investigation. Integration maintenance sits with the platform, and Probing keeps measuring agreement with analysts after every change, so drift shows up in the numbers before it shows up in a missed attack.
A trusted verdict is not the end goal. It is what lets the SOC hand the AI more responsibility, one step at a time, and turn investigations into a better defense.
The first step is controlled response. With Bucketing, alerts are grouped by type and the AI earns autonomy one type at a time. Where Probing shows sustained agreement with analysts, a team can let Qevlar act within its permissions and change process, with its own confidence thresholds. Qevlar then verifies whether the action worked. Where trust isn't proven yet, people stay in charge.
The second is a learning loop. An investigation that ends in a closed ticket is wasted. Every investigation that resolves the same way, or exposes something no rule covered, is evidence detection engineers can use to tune, retire or write rules. Static detections age the day they're written; a defense fed by its own investigations keeps changing as attackers do.
The third is looking beyond the rules entirely. Watchers hunt continuously in the telemetry a team already collects, so a vulnerability that can't be patched this quarter, or a privileged account that starts behaving differently, stays under watch until the fix lands. None of this is safe to switch on until analysts trust the verdicts underneath it.
A team that wants analysts to act on AI verdicts is building three things at once: a system that investigates, a layer that makes its investigations reproducible, and a way to measure, continuously and blind, whether its conclusions match those of the people who will rely on them.
The first is what a pilot demonstrates. The other two decide whether the pilot ever leaves validation, and whether the SOC can ever tell its board, with evidence, that it is defended rather than just busy.
Teams that start building often reach a similar conclusion about where their effort belongs. Vancord, which runs a SOC for around 125 customers, began building its own AI investigation system before choosing Qevlar, and stopped because of cost at the scale of one company, constantly changing models, and the need for ML engineers to backtest everything.
"Our goal is to secure customers, not build systems. We would have just ended up being a developer shop."
Matthew Fusaro, Head of Security Operations, Vancord
