Evaluating Agentic Output Before GA

@ SentinelOne

Video of Purple AI Agentic Investigation

Overview

This was an evaluative research initiative I led at SentinelOne for Purple AI Auto Investigation — a feature that lets an analyst launch a complete agentic investigation with a single click, with the agent gathering cross-stack evidence, building an attack timeline, and returning an explainable verdict.

This was the company's first major agentic feature to reach the market. The research ran as a five-week beta program with customers using the feature in their own production environments, ahead of general availability.

Stakeholders: Director of PM for AI, Lead Product Designer, Engineering Leads

Evaluating something that doesn't have a "correct screen"

Most usability work evaluates an interface a person navigates. Here, the thing under evaluation was the output of a reasoning agent — a written investigation summary whose content changed with every alert, every environment, and every run.

That split the research into two questions the team needed answered before GA:

  1. Is the output good? Is the agent's evidence-gathering accurate, is the verdict defensible, and does an analyst trust it enough to act?

  2. Is the output legible? Is the summary structured and laid out so an analyst can find what they need without re-reading it?

Getting this wrong in either direction was expensive. An agent that reasons well but presents poorly gets ignored. An agent that presents beautifully but reasons badly gets trusted when it shouldn't be — which, in security operations, is the worse failure.

Considerations

Real environments, real customer data

Live alerts, with bugs that could surface mid-session

A narrow eligible population

Right SKU, alert volume and limited time with busy analysts

Non-deterministic output

No fixed artifact to test against. Different organizations → Different alerts → Different outcomes

Method

We ran a structured beta with five customers, conducting bi-weekly sessions and a running issues list.

Findings

The shape of the issues list was itself the finding. The most useful signal wasn't any single issue — it was the distribution of them.

The majority of what customers raised turned out to be feature requests, not defects in data accuracy or summary quality. For a first agentic feature, that inversion was the result we wanted: it meant the core implementation was working, and customers had moved past "is this correct?" to "here's what else I want it to do."

Reading the issues list as a dataset, rather than a backlog, is what turned five weeks of scattered feedback into a GA-readiness signal.

The automation tension: what customers asked for vs. what the button was protecting

CUSTOMERS ASKED FOR

Run investigations automatically

Saves time, eliminates need to click and wait for results

VS

THE CLICKED PROTECTED

A gate on high and critical alerts

Every run costs $$$, at hundreds of alerts a day

Is this usable?

What’s the right unit of automation, and what does it cost at scale?

Research Impact

The beta validated GA readiness

Purple AI Auto Investigation reached general availability at RSAC 2026 in March 2026. Notably, it shipped with the one-click model intact — a single click to launch, with analyst-in-the-loop governance — rather than the fully automatic behavior customers had asked for. The severity gating the beta examined held through to launch.

Reframed the problem for the team

The research moved the internal conversation from output quality to operating cost. Product managers had to now take into consideration the cost per agentic investigation into pricing models as the price to run every single investigation is costly.  

Established how the team runs and reports on Betas

Other teams adopted the issue categorization method
The two-level structure — main themes, then sub-topics that could be individually prioritized — was picked up by other teams running their own betas.

We introduced a post-beta recap for participants
We closed the loop with the customers who gave us their time — showing them how their feedback had been gathered and categorized, and what we intended to do with it.

Learnings

Beta feedback is a dataset, not a queue

  • The category distribution told us more about readiness than any individual issue did

  • Categorizing early, rather than at the end, made the pattern visible while there was still time to act on it

Researching agentic features means researching trust, not just usability

  • With non-deterministic output, "did they complete the task" is the wrong measure — "did they believe it, and what did they do to check" is the right one

  • The moment an analyst leaves the summary to go verify something manually is the highest-value observation in the session

Customer requests are signals, not specs

  • Every customer asked for automatic investigations; none of them were pricing the compute

  • Part of the job was translating a unanimous feature request into the tradeoff the team actually needed to decide