The ability to perform A/B split tests within your SMS program will provide valuable insight into the type of content or the time of day that drives engagement or increases conversion rates to your entire SMS list.
There are numerous strategies you can employ in a content-type split test, consider one of the following options:
Attachment vs No attachment (SMS vs MMS)
Word Order
Personalization vs No Personalization
Calls-to-Action
Message Content-Length
Subject Lines
Subject Line vs No Subject Line
Images: People vs Products
Images: with Copy vs Image-only
Images: Static vs Animated GIF
Incentive vs No Incentive
Offer Different Incentives
Slang or Abbreviations vs Spelled-out Words
Capitalization vs Lowercase
Hard Returns vs One block of copy
Time of Day
⚠️ Data Acquisition SMS messages are not available as a split test option.
Set-up
To create a split test, follow the steps below:
Navigate to Messages > SMS SMS Broadcast > click New Campaign button.
💡 If you have multiple sender codes, you will need to select the correct sender code from list navigation dropdown menu.
Select the type of split test.
Split Test Content (A/B)
Split Test Send Time (A/B)
Next configure the Setup, Compose, Split Test Settings, and Review & Send components of the split test.
Setup
The setup consists of selecting the list, choosing an audience, and setting both Listrak and Google Analytics to the split test.
Enter a unique name for the split test.
💡 Clicking Save at this point (located at the bottom of the control panel), will save your work and then update the panel to Edit Split Test.Fill out the remaining fields as you would when creating a Broadcast Campaign. Click here for instructions on completing the Setup configurations.
Click Continue to move to the next step.
💡 Click the Step name in the left sidebar to configure it at any time while in the split test builder. The split test does not need to be built in a specific order.
Compose
Variation Setup
Once the strategy has been determined, the next step is to set up the number of variations (up to 5) and apply the message content to each by following these steps:
Configure the first Variation (A) by clicking Edit. Follow the same steps as you would when creating a Broadcast Campaign Message.
Add the additional message variations:
For a Content Split Test
Want to test similar content?
Click Duplicate within the Actions icon to duplicate a variation. Then click Edit to make small changes - like adding a link, attaching a file, or adding personalization.Want to test different content?
Select Add Variation for each message variation, click Edit to configure.
For a Send Time Split
Select Add Variation for each number of messages (or different times) you will be testing. The message configured in Variation A will automatically populate in each additional variation.
⚠️ This is different than how Add Variation works in a Content Split Test setup.
When all variations have been added, click Continue to move to the next step.
💡 Helpful Tips for Composing a Split Test
Need to remove a variation?
Click Delete within the Actions icon to remove a variation.
Need to send a test message for any of the variations?
Simply click Send Test within the Composition panel of the variation you would like to test. ⚠️ A test list must be set up to perform this function. Learn more about how to set up a list here.
Split Test Settings
The next step is to choose the testing criteria and the duration of the test.
Enter a description of the Split Test (100 characters max) - Use a description that will best summarize the type of test you will be running.
Select the Winning Metric - Choose either Click-Through-Rate or Conversion Rate. The type of content being tested will help to determine the winning metric, reference the examples below when setting up the split test:
Click-Through-Rate (Percentage of customers who click on a link) -
Images variations, Images vs Text, Calls-to-Action, Headline, Slang or
Abbreviations vs Spelled-out Words or, Message Content-Length.Conversion Rate (Number of conversions divided by the number of site visits) - Incentive vs No Incentive, Offering Different Incentives, Word Order, Headlines, Calls-to-Action, or Attachment vs No Attachment.
Configure the Variation Distribution - Choose to maintain the same distribution ratio for each variation (default) or send the winning message to a remainder group.
Ratio distribution option:
This default setting is automatically calculated based on the number of variations in the split test.
2 Variations (minimum requirement) - 50 / 50
3 Variations - 33 / 33 / 33
4 Variations - 25 / 25 / 25 / 25
5 Variations - 20 / 20 / 20 / 20 / 20
Set the duration by entering the number of Days (range 1-7) or Hours (max 168) to run the test.
Winning message to the remaining group option:
Click the box next to the option to enable.
Configure the size of the remainder group by sliding the bar in that row.
💡 The calculations for the variations will be redistributed equally based on the ratios listed above.
Set the duration - Choose by ending the test after a number of days/hours (default) or by setting a specific value based on the Winning Metric chosen in Step 2.
To specify a metric, click the box next to the option to enable.
Set the number of days/hours
Set the winning metric value
Click Continue to move on to the next step.
Review & Send
On the Review & Send page confirm the various settings, make any last-minute edits and schedule the test.
Editing the Split Test
Click on the Edit button to update any of the configuration options or message variations.
Schedule the Send
For a Content Split Test
Choose the send date and send time (EST).
You can select 'Time Zone Optimized' to send at the specified time in a contact's time zone. You will be able to see how many contacts received messages in each time zone when viewing reporting on the individual message. Messages must be scheduled at least two hours in advance when using this feature.
For a Send Time Split Test
Simply select different send times (EST) for each variation.
⚠️ The Time Zone Optimized feature is either on or off for all variations.
Confirm and schedule
The last step to complete the split test setup is to confirm all the settings, variations, and scheduled date/time, to do so click Confirm and the bottom of the page.
Reporting
The SMS Split Test Dashboard displays important information ranging from the test status, the winning variation, revenue earned, and more about each split test created on your sender code. Access this dashboard by navigating to Messages > SMS SMS Split Test.
Split Test Dashboard
New Split Test: Click to create a new test.
Search: Use the name to find a split test on the dashboard.
Name: The name of the split test. Click the name to view the report dashboard for the individual split test.
Send Date: The start date and time of the split test.
End Date: The end date and time of the split test.
List Name: The name of the list the split test was sent to.
Status: Displays the current status of the test.
Draft - test is still being configured.
In Progress - test is in progress.
Scheduled - test is scheduled to begin in the future.
Completed - test is complete with results available.
Actions: Click the icon and select an option. These vary based on the status of the test:
Actions | Definition | Available Status |
Edit | Allows you to make changes to any part of the split test. | Draft |
Duplicate | Allows you to make a copy of the split test. | Draft, Scheduled, In Progress, or Completed |
Delete | Allows you to remove a split test. | Draft or Scheduled |
View Report | Allows you to access the individual split test report dashboard. | In Progress or Completed |
Individual Split Test Reporting Dashboard
Test Details
Split Test Name (required): The name of the split test.
Export: Click to export the split test results (Excel file format).
Split Test Description (optional): The name used to describe the content/goal of the Split Test.
Number of Variations: Shows number of variations in the test (minimum 2, max 5).
Test Type: Indicates the type of test - Content.
Campaign Name: The name given to identify the split test in your Google Analytics dashboard.
List: The name of the list the split test was sent to.
Filters Applied (optional): Displays any filters applied to this test.
Status/Date: The current status of the test and duration of the test (date/time).
Winner: The name of the winning variation.
Winning Metric: The metric that determined the winning variation.
Message Variation Details
Click the tab associated with each variation to see the message details.
Message Type: Indicates the type of message - SMS or MMS.
Tracking Compliance: Indicates if the compliance messaging was pasted into the message.
Revenue:
Total Revenue: The total amount of revenue generated by the message.
Average Order Value (AOV): The average order value of orders placed as a result of the message.
Conversions:
Count: The total number of conversions made from the message.
Rate: The percentage is calculated by the total number of messages sent divided by the total number of conversions made.
Click-Through:
Count: The total number of links clicked.
Rate: The percentage is calculated by the total number of messages sent divided by the total number of links clicked.
Split Test Messages
This section displays the metrics for every variation of the split test so you can compare the performance of each message.
Name: The variation name applied to the test (minimum 2, max 5).
Winning Variation: The winning variation is marked with the status of 'Winner'.
Type: Indicates the type of message - SMS or MMS.
Send Date: The start date and time of the split test.
Status: Displays the current status of the test.
In Progress - test is in progress.
Scheduled - test is scheduled to begin in the future.
Completed - test is complete with results available.
Sent: The number and percentage of messages sent of the variation.
Delivery Rate: Percentage of the number of messages sent of the variation compared to total messages sent.
Failed Rate: Percentage of the number of messages failed to be delivered of the variation compared to total messages sent.
Capped: The number of messages not sent to contacts because they had met the frequency capping threshold.
Click-Through: The number and the percentage of links clicked.
Conversions: The number and percentage of conversions made by the message.
Revenue: The total amount of revenue generated by the message.
Average Order Value (AOV): The average order value of orders placed as a result of the message.