From Testing Purgatory to Go-Live: Breadth and Depth for AI Agents

You can test your agent forever. But how do you know when you are ready to go live?

Testing is critical. Without it, you won’t have confidence in your agent’s ability to handle real-world scenarios. But with hundreds or thousands of different customer queries, teams often think you have to test everything. They get stuck in the painful place we call testing purgatory.

We believe the solution isn’t more tests—it’s testing the right things in the right order. Testing can be an elegant, simple, and fun process.

From infinite to finite

Customer requests can be written an infinite number of ways:

• I was charged by mistake.
• I want this returned and refunded.
• Can I get my $47.23 back?
• Please issue a refund for this order.

All of these queries fall into one bucket: refunds. We call these intents—based on the specific action or question that needs to be answered or resolved. This reframes the problem of testing—instead of testing every possible question, we test intents, systematically.

We break this into three phases: breadth, depth, edges.

Breadth, depth, edges:

Breadth: The goal here is coverage, to see how the agent performs across all intents. This usually means tagging historical conversations and testing with 5-10 scenarios for each intent.

Depth: Not all intents are valued the same. Spending more time on how the agent handles cancellations might be more worthwhile than testing every way a user could ask about the shipping policy. The goal here is to prioritize the highest-volume and highest-risk intents and to test variations within each: different products, timing, customer context.

Edges: In an ideal world, our tests are an exact match with reality. But sometimes customers have unexpected requests. Here, we test adversarial users, missing data, and ambiguous requests. We make sure we define when the agent can handle the request vs. when it should escalate to your support team.

What makes a good test

Does coverage across all your intents mean confidence? Not quite.

You can have 100 tests that all pass but don't tell you how your agent will perform in the real world. Coverage alone does not lead to confidence. So in addition to coverage, good tests use real language, include ambiguity and missing context, and actually stress your agent against what it might encounter once live.

A real example

Let’s see how this works with an example. Imagine we are working with Frost & Feast, a direct-to-consumer meal brand delivering flash-frozen, chef-crafted, high-protein meals. We are launching a support agent that can handle refunds, issue replacements, and answer general product Q&A.

Week 1 - Breadth

In week 1 we set up 100 test scenarios across 30 different intents, including returns, exchanges, refunds, order status, and policy questions. We also identify the five intents that get the most traffic and which are the highest value to optimize.

Weeks 2/3 - Depth

We decide order status is a critical workflow to set up for the agent since our product is frozen. When customers reach out, they need to get precise estimates of when it will be shipped and delivered.

We might test different products, different time windows since the initial purchase, different tones of customer requests, different zip codes and locations. As we test, we make adjustments and improve the agent, until we feel confident in our most important workflows.

Week 4 - Edges

To build full confidence in the agent, we test the unexpected. We test returns outside the policy window, damaged items, very angry customers, and adversarial customers. We define cases where the agent should respond, and when it should escalate to the human team.

Day 30 - Launch

By launch date we have full confidence in our agent. We haven’t tested *everything—*which is the point. Instead, we tested systematically. We’ve tested and evaluated our agent across the variety of intents it can handle in the real world.

Towards systematic testing

Most teams get stuck in testing loops that drag on for months. Eventually they wonder: is this even worth it?

The solution isn’t more tests: it’s testing the right things in the right order.

If you’re looking for help going live with your agent, we’d love to help.

→ Get a demo