Applied LabsDocs
Skip to Content

Set Up an Experiment

Experiments help you compare two or more versions of an agent on real traffic so you can confidently decide what to ship.

This guide walks you through planning, launching, and monitoring an experiment.

Before you start

  • Decide what change you are testing. One change per experiment is best.
  • Decide what success means. Pick one primary metric.
  • Make sure the outcome you care about is already showing up in your analytics today.

1. Create the experiment

  1. Open your agent.
  2. Go to Deploy -> Experiments.
  3. Click New experiment.

Fill in:

  • Name — a clear label your team will recognize
  • Identifier — a short, stable name for reporting
  • Description — what you are changing and why
  • Dates — optional start and end windows

2. Add variants

Variants are the versions of your agent you want to compare.

  1. Add at least two variants.
  2. Choose one variant as Control.
  3. Set the weight for each variant.

Recommendations:

  • Start with a 50/50 split for two variants unless you are being cautious.
  • Keep variant names simple, such as control and treatment.

3. Choose what to measure

You will choose:

  • Primary metric — the main definition of success
  • Secondary metrics — optional context and trade-offs
  • Guardrails — optional “do not let this get worse” metrics

4. Send traffic to the experiment

Creating an experiment does not automatically send traffic to it.

  1. Go to Deploy -> Traffic control.
  2. Create a new rule.
  3. Choose who is eligible.
  4. Choose a starting percentage.
  5. Choose your experiment as the destination.

Tips:

  • Start small, such as 5 to 10 percent, and expand once everything looks healthy.
  • Keep rules simple so results are easier to interpret.

5. Start the experiment

When everything is ready:

  1. Return to Deploy -> Experiments.
  2. Open your experiment.
  3. Change the status to Running.

Once it is running, you should see exposures begin to accumulate.

6. Stop the experiment and ship the winner

When you are ready to end the test:

  1. Change the status to Completed.
  2. Update your routing so all traffic goes to the winning version.
  3. Keep the experiment for reference so your team can learn from it later.

Example: conversion experiment

This is a common pattern when success is defined as more visitors completing a key action after engaging with your agent.

What to test

Pick one change to test, such as:

  • A shorter first message versus a longer first message
  • A different recommendation or offer
  • A different call-to-action wording

Example metric setup

Good primary metric examples include:

  • Purchase conversion
  • Checkout start conversion
  • Click-through conversion

Add one to three secondary metrics for context, such as revenue, average order size, or escalation rate.

Example rollout plan

  • Start at 10 percent for the first day
  • Increase to 25 percent once things look healthy
  • Increase to 50 percent once you are confident

Troubleshooting

I do not see any exposures

  • Confirm your routing rule is active and ordered correctly.
  • Confirm the audience definition is not too restrictive.
  • Wait a few minutes. New traffic is required before you will see exposure counts.

I see exposures, but not the outcome I care about

  • Confirm the outcome is tracked and visible outside of experiments first.
  • Run a real end-to-end test flow and confirm the outcome appears in analytics.

Next steps

Last updated on
Managing Agent VersionsUnderstanding Results