An A/B Test lets you send two versions of a Message Broadcast and use real data to confirm which one performs better. Compare two variations with different copy or messaging angles, then apply the winning pattern to your next broadcast.
What an A/B Test Is
An A/B Test is a feature that splits the same broadcast into two patterns and compares metrics like open rate and click rate.
When you set different messages for Pattern A (main) and Pattern B (sub) and send them, the results for each are tallied. The one with the higher numbers is shown as the "winning pattern," so you can decide which direction to improve based on data rather than gut feeling.
Once you understand what changes drive a different response, your next broadcast becomes more precise. We recommend letting the Marketing Agent handle the analysis while you use the A/B Test as the basis for your decisions.
Let the Agent Handle the Analysis
You can consult the Marketing Agent about comparing broadcast results and planning your next move.
In Auto mode or Consult mode, the agent organizes "which pattern worked" and "what to try next" based on your past broadcast data. After the A/B Test results are in, ask "how should I read these results?" and the agent will suggest what the winning pattern tends to look like and how to apply it next time.
You can choose whether to proceed with what the agent proposes using the approval action. If you are satisfied with the content, select "Approve"; if you want to hold off, select "Cancel". If you would rather configure the details yourself, use the manual steps in the next section.
Create an A/B Test
You create an A/B Test from the Message Broadcast editing screen.
- Create a new broadcast with "Create Broadcast", or open an existing broadcast
- From "More actions" on the editing screen, select "Create A/B Test"
- In the confirmation dialog, select "Create". The original broadcast content is saved as a draft and becomes Pattern A
- Switch the view to Pattern B and edit the message content you want to compare
- Review the content, then schedule or send the broadcast
The Roles of Pattern A and Pattern B
Pattern A and Pattern B have different editable items. Shared broadcast conditions are managed on the Pattern A side.
| Pattern | Role | Editable items |
|---|---|---|
| Pattern A (main) | The baseline broadcast | Name, message, broadcast settings, recipients, sender |
| Pattern B (sub) | The other side being compared | Name, message |
Broadcast settings such as recipients, send time, and sender are configured in Pattern A and applied to both patterns in common. In Pattern B, you can only change the name and the message.
What You Can Test
An A/B Test compares differences in the message itself. Because recipients and send timing are shared across both patterns, what you are validating is mainly the "content."
| Test target | Comparison example |
|---|---|
| Message copy | Casual tone vs. formal tone |
| Messaging angle | Emphasizing exclusivity vs. emphasizing value |
| Message type | Text vs. Rich Message |
| How images and buttons are shown | Rich Message with an image vs. text only |
Limiting each test to a single changed element makes the results easier to interpret. If you change multiple elements at once, you cannot tell which difference drove the outcome.
Check the Results
You check A/B Test results on the broadcast detail screen.
After sending, the numbers for each pattern are tallied, and the one with the better response is shown as the "winning pattern." If there is no difference between the two, it is shown as a "draw."
| Metric | Description |
|---|---|
| Delivery rate | The percentage of messages that were delivered |
| Open rate | The percentage of messages that were opened |
| Click rate | The percentage of links in the message that were clicked |
| Winning pattern | The pattern that performed better when the metrics are compared |
You can also review the results in a per-pattern table that shows the breakdown of figures such as number sent, number delivered, number opened, and number clicked.
Cancel an A/B Test
You can turn an A/B Test you created back into a regular Message Broadcast.
From "More actions" on the editing screen, select "Cancel A/B Test" and proceed through the confirmation dialog to revert to a regular broadcast that inherits the Pattern A content. At this point, the content you entered in Pattern B is deleted. If you want to keep the Pattern B content, duplicate it as a separate broadcast before canceling.
When Things Do Not Go as Planned
Here are the points where people commonly get stuck with A/B Tests, along with how to resolve them.
- No difference appears in the results: The difference between Pattern A and Pattern B may be too small. Rather than just tweaking the wording at the end of sentences, try changing elements that are likely to affect the response, such as the messaging angle or the message type.
- There are items you cannot edit in Pattern B: In Pattern B, you can only edit the name and the message. When you want to change recipients, send time, or sender, switch to Pattern A and configure them there.
- Not enough to base a decision on: When the recipient pool is small, it is hard to tell whether a difference is luck or real performance even when one appears. Testing with a broadcast that reaches enough people improves the reliability of the results.