An A/B Test lets you send two patterns of a Message Broadcast and use real data to confirm which one performs better. Compare two variations with different copy or messaging angles, then apply the winning pattern to your next broadcast.
What is an A/B test?
An A/B Test is a feature that splits the same broadcast into two patterns and compares metrics like open rate and click rate.
When you set different messages for Pattern A (main) and Pattern B (sub) and send them, the results for each are tallied. The one with the higher numbers is shown as the "Winning pattern", so you can decide which direction to improve based on data rather than gut feeling.
Once you understand what changes drive a different response, your next broadcast becomes more precise. We recommend letting the Marketing Agent handle the analysis while you use the A/B Test as the basis for your decisions.
Letting the agent handle the analysis
You can consult the Marketing Agent about comparing broadcast results and planning your next move.
In Auto mode or Consult mode, the agent organizes "which pattern worked" and "what to try next" based on your past broadcast data. After the A/B Test results are in, ask "how should I read these results?" and the agent will suggest what the winning pattern tends to look like and how to apply it next time.
You can choose whether to proceed with what the agent proposes using the approval action. If you are satisfied with the content, select "Execute"; if you want to hold off, select "Not now". If you would rather configure the details yourself, use the manual steps in the next section.
Creating an A/B test
You create an A/B Test from the Message Broadcast editing screen.
- Create a new broadcast with "Create Broadcast", or open an existing broadcast
- From "Other actions" on the editing screen, select "Create A/B test"
- In the confirmation dialog, select "Create". "Pattern A" and "Pattern B" tabs appear in the message settings on the same page, and the original content becomes Pattern A
- Switch to the "Pattern B" tab and edit the message content you want to compare
- Review the content, then schedule or send the broadcast
The tabs switch only the message content and the preview. You can move between them freely, even before saving. A test send delivers the pattern you're currently viewing.
Per-pattern settings and shared settings
The only thing that changes per pattern is the message content. Everything else is shared by both patterns.
| Setting | How it works |
|---|---|
| Message content | Per pattern. Switch between the "Pattern A" and "Pattern B" tabs in the message settings to edit each one. The preview follows the content of the selected pattern |
| Name, folder, broadcast settings, recipients, sender, UTM settings | Shared. Set them once outside the tabs, and they apply to both patterns |
You give the test one shared base name. When you save, "(A)" and "(B)" are added to each pattern automatically.
What you can test
An A/B Test compares differences in the message itself. Because recipients and send timing are shared across both patterns, what you are validating is mainly the "content."
| Test target | Comparison example |
|---|---|
| Message copy | Casual tone vs. formal tone |
| Messaging angle | Emphasizing exclusivity vs. emphasizing value |
| Message type | Text vs. Rich Message |
| How images and buttons are shown | Rich Message with an image vs. text only |
Limiting each test to a single changed element makes the results easier to interpret. If you change multiple elements at once, you can't tell which difference drove the outcome.
Checking the results
You check A/B Test results on the broadcast detail screen.
After sending, the numbers for each pattern are tallied, and the one with the better response is shown as the "Winning pattern". If there is no difference between the two, it is shown as "Tie".
| Metric | Description |
|---|---|
| Delivery rate | The percentage of messages that were delivered |
| Open rate | The percentage of messages that were opened |
| Click rate | The percentage of links in the message that were clicked |
| Winning pattern | The pattern that performed better when the metrics are compared |
You can also review the results in a per-version table that shows the breakdown of figures such as number sent, number delivered, number opened, and number clicked.
Canceling an A/B test
You can turn an A/B Test you created back into a regular Message Broadcast.
From "Other actions" on the editing screen, select "Cancel A/B test", then select "Cancel A/B test" in the confirmation dialog. The tabs disappear, and the broadcast reverts to a regular one that inherits the Pattern A content. At this point, the content you entered in Pattern B is deleted. If you want to keep the Pattern B content, duplicate it as a separate broadcast before canceling.
When things don't work
Here are the points where people commonly get stuck with A/B Tests, along with how to resolve them.
- No difference appears in the results: The difference between Pattern A and Pattern B may be too small. Rather than just tweaking the wording at the end of sentences, try changing elements that are likely to affect the response, such as the messaging angle or the message type.
- You can't find the broadcast settings in Pattern B: The name, recipients, send time, and sender aren't per-pattern settings. They live outside the tabs as shared settings, and changing them applies to both patterns.
- Not enough to base a decision on: When the recipient pool is small, it is hard to tell whether a difference is luck or real performance even when one appears. Testing with a broadcast that reaches enough people improves the reliability of the results.