Evaluating Agentic Output Before GA

Considerations

Real environments, real customer data

We tested in customers' live production environments against their actual alerts. This was non-negotiable — synthetic data can't tell you whether an agent's reasoning holds up against a real, messy intrusion — but it meant unpredictable bugs and unexpected agent behavior surfacing mid-session, in front of the customer.

A narrow eligible population

Participation required a specific SKU. That collapsed the recruiting pool from "SentinelOne customers" to a small set of organizations who had the right entitlement, the right alert volume, and analysts with time to give us.

Non-deterministic output

There was no fixed artifact to test against. Two analysts running the same feature saw different investigations. Standard task-success measures don't transfer cleanly when the stimulus changes every time — the research design had to accommodate that.

Method

We ran a structured beta with five customers who had the required SKU, using the feature against live alerts in their own environments over five weeks.

The format was bi-weekly sessions to walk through feedback, which fed a running issues list — every piece of customer input logged, categorized, and triaged with the product and engineering team.

Sample Agenda:

1. Recap of usage

2. Going through specific notes and feedback

3. Prep for next call

At the end of the Beta, we asked them to fill out an ‘End of Beta’ survey which captured overall satisfaction, SUS score, learnability, usefulness, and efficiency (was time saved?).

Findings

The shape of the issues list was itself the finding

The most useful signal wasn't any single issue — it was the distribution of them.

The majority of what customers raised turned out to be feature requests, not defects in data accuracy or summary quality. For a first agentic feature, that inversion was the result we wanted: it meant the core implementation was working, and customers had moved past "is this correct?" to "here's what else I want it to do."

Reading the issues list as a dataset, rather than a backlog, is what turned five weeks of scattered feedback into a GA-readiness signal.

Five categories of feature requests

1. Summary and content enhancement

2. Summary formatting

3. Workflow, actions, & automation

4. Core investigation performance and accuracy

5. UI, navigation, and accessibility

The automation tension: what customers asked for vs. what the button was protecting

The most consistent request across customers was to make investigations run automatically instead of requiring a click.

The click was deliberate. It existed to gate auto-investigation to high and critical severity alerts — because running full agentic investigations against the hundreds of alerts a SOC receives daily carries real cost implications. The button was a cost control disguised as an interaction.

This reframed a chunk of the team's work. The question stopped being "is this usable" and became "what is the right unit of automation, and what does it cost to run at production volume?" — an efficiency-versus-cost tradeoff the team had to resolve before scaling the feature.

Surfacing that tension early, with evidence from five real customers rather than one loud one, is what let the team treat it as a design and pricing problem instead of a bug report.

Research Impact

The beta validated GA readiness

Purple AI Auto Investigation reached general availability at RSAC 2026 in March 2026. Notably, it shipped with the one-click model intact — a single click to launch, with analyst-in-the-loop governance — rather than the fully automatic behavior customers had asked for. The severity gating the beta examined held through to launch.

Reframed the problem for the team

The research moved the internal conversation from output quality to operating cost. Product managers had to now take into consideration the cost per agentic investigation into pricing models as the price to run every single investigation is costly.  

Established how the team runs and reports on Betas

The process outlived the project in two directions.

1. Other teams adopted the issue categorization method
   The two-level structure — main themes, then sub-topics that could be individually prioritized — was picked up by other teams running their own betas.

2. We introduced a post-beta recap for participants
   We closed the loop with the customers who gave us their time — showing them how their feedback had been gathered and categorized, and what we intended to do with it.

A note on outcome metrics

My role was eliminated in the May 2026 layoffs before post-GA adoption data was available, so this write-up ends at launch rather than at usage numbers.

Had I stayed, the measures I'd defined for the next phase were:

• Investigation acceptance rate — how often analysts act on the agent's verdict without independently re-investigating. The trust measure, and the one that matters most for an agentic feature.

• Manual verification rate — how often analysts pivot to raw logs or another system after reading the summary. A direct read on whether the output is complete.

• Cost per investigation against severity tier — to test whether the severity gate was drawn in the right place, and what a broader automation threshold would actually cost.

Baseline for the first two would have come from beta session observation; the third would have needed instrumentation the team was already scoping.