A/B testing, also called split testing, lets you compare campaign variants to determine which version achieves the best results.
In Dotdigital Marketing you can test email campaigns, SMS campaigns, subject lines, sender names, from addresses, and campaign content to improve engagement and optimise future campaigns.
How A/B testing works in Marketing
Key components of A/B testing:
Variants
Different versions of your campaign
Audience split
Contacts are randomly divided into groupsPerformance metrics
Such as opens, clicks, or ROIWinner selection
Automatic or selected by you
A/B testing allows you to validate changes on a smaller audience before sending the campaign more widely.
A/B testing compares multiple versions (variants) of a campaign.
Each contact receives only one variant of a campaign.
Variants are created and managed within a single campaign, not as separate campaigns.
Each variant is sent to a portion of your audience, based on how your test is configured. Marketing measures how recipients interact with each version and provides performance data to identify the best-performing variant.
Depending on how you configure your test, Marketing can:
send test variants to a sample audience and automatically send the winning variant to the remaining contacts
split your full audience evenly across all variants
allocate a defined percentage of contacts to a control variant and distribute the remaining audience across test variants
How audience allocation works
When you run an A/B test, Marketing divides your audience into groups.
The exact allocation depends on the type of test you choose, but in all cases:
Contacts are randomly assigned to a variant
Each contact receives only one version of the campaign
Variants are distributed evenly within each test group
Example:
If you test two variants on 10% of an audience of 10,000 contacts, each variant is sent to 500 contacts during the test phase. The remaining 9,000 contacts may receive a winning variant, depending on the split test type you choose.
The way your audience is divided and how a winner is selected relies on the split test type you choose.
What you can test
The elements you can test depend on the campaign channel.
Email campaigns:
Subject lines
Sender names
From addresses
Campaign content
SMS campaigns:
Campaign content
For step-by-step instructions, see:
Split test types
Marketing provides three ways to run A/B tests. Each method supports different goals.
1. Test then send winner
Send each variant to a portion of a test sample of your audience, split evenly across variants. You define both the test audience size and the duration of the test period. After the test period ends, Marketing selects a winner and sends it to the remaining contacts. You can allocate up to 60% of your audience to the test phase and define how long the test runs before a winner is selected.
Key characteristics:
Test runs on a percentage of your audience that you define during setup.
Marketing selects a winner
Remaining audience receives the winning variant
2. Split audience equally
Divide your full audience evenly across all variants.
Use this when you want a fair comparison between versions.
Key characteristics:
No automatic winner send
Each variant receives the same audience size
There is no automatic winner selection. Review the split test report metrics and determine the best-performing variant based on your campaign goals.
3. Set control variant
Choose one version as a control, and compare other variants against it.
Use this when you want to test changes against a baseline.
Key characteristics:
You define what percentage of your audience receives the control variant.
Remaining audience is split across test variants.
There is no automatic winner selection. Use the split test report to compare the control against the test variants and determine the best-performing version.
Choose the right test type
Use the table below to guide your decision:
Goal | Recommended test type |
Maximise performance automatically | Test then send winner |
Compare variants evenly | Split audience equally |
Measure impact against a baseline | Set control variant |
Key behaviours and limitations
Be aware of the following when using A/B testing:
A/B testing is only available for standard campaigns.
Contacts are randomly allocated to variants.
You can create up to 10 variants.
Once a campaign starts sending, you cannot edit it.
Test and preview variants
Before sending your campaign, you can preview or test specific variants.
Some behaviours depend on the type of split test you configure:
When testing creative variants, you can select which version to preview.
Reporting and results
After your campaign sends, you can review A/B test performance in your reports.
To access your report:
In Marketing, go to Analytics > Reports.
Select your campaign.
Marketing provides a split test report that compares all variants side by side.You can:
View performance metrics for each variant in a single table.
View individual variant reports using the campaign report filters.
Compare percentage metrics and total metrics.
Export report data as a CSV file.
Open a preview of each variant.
The report helps you understand which version performed best based on your chosen metric.
Reporting behaviour and limitations
Be aware of the following when analysing results:
Winning and losing variants
For Test then send winner tests, you can view both winning and losing variants.
The option to show losing variants is only available for this test type.
Revenue tracking
Revenue is reported at campaign level only.
You cannot view revenue per variant.
Tie scenarios
If no winner can be determined, the first variant created is used.
This can happen if no engagement occurs during the test period.
Heatmaps and link hotspots
These show combined engagement for the full campaign.
They are not broken down by variant.