Skip to main content
SMS Split Testing

Learn how to create a split test for SMS broadcast messages.

Support avatar
Written by Support
Updated over 3 months ago

The ability to perform A/B split tests within your SMS program will provide valuable insight into the type of content or the time of day that drives engagement or increases conversion rates to your entire SMS list.

There are numerous strategies you can employ in a content-type split test, consider one of the following options:

  • Attachment vs No attachment (SMS vs MMS)

  • Word Order

  • Personalization vs No Personalization

  • Calls-to-Action

  • Message Content-Length

  • Subject Lines

  • Subject Line vs No Subject Line

  • Images: People vs Products

  • Images: with Copy vs Image-only

  • Images: Static vs Animated GIF

  • Incentive vs No Incentive

  • Offer Different Incentives

  • Slang or Abbreviations vs Spelled-out Words

  • Capitalization vs Lowercase

  • Hard Returns vs One block of copy

  • Time of Day

⚠️ Data Acquisition SMS messages are not available as a split test option.


Set-up

To create a split test, follow the steps below:

  1. Navigate to Messages > SMS SMS Broadcast > click New Campaign button.

    💡 If you have multiple sender codes, you will need to select the correct sender code from list navigation dropdown menu.

  2. Select the type of split test.

    1. Split Test Content (A/B)

    2. Split Test Send Time (A/B)

  3. Next configure the Setup, Compose, Split Test Settings, and Review & Send components of the split test.

Setup

The setup consists of selecting the list, choosing an audience, and setting both Listrak and Google Analytics to the split test.

  1. Enter a unique name for the split test.
    💡 Clicking Save at this point (located at the bottom of the control panel), will save your work and then update the panel to Edit Split Test.

  2. Fill out the remaining fields as you would when creating a Broadcast Campaign. Click here for instructions on completing the Setup configurations.

  3. Click Continue to move to the next step.

💡 Click the Step name in the left sidebar to configure it at any time while in the split test builder. The split test does not need to be built in a specific order.


Compose

Variation Setup

Once the strategy has been determined, the next step is to set up the number of variations (up to 5) and apply the message content to each by following these steps:

  1. Configure the first Variation (A) by clicking Edit. Follow the same steps as you would when creating a Broadcast Campaign Message.

  2. Add the additional message variations:

    1. For a Content Split Test

      1. Want to test similar content?
        Click Duplicate within the Actions icon to duplicate a variation. Then click Edit to make small changes - like adding a link, attaching a file, or adding personalization.

      2. Want to test different content?

        Select Add Variation for each message variation, click Edit to configure.

    2. For a Send Time Split

      1. Select Add Variation for each number of messages (or different times) you will be testing. The message configured in Variation A will automatically populate in each additional variation.

        ⚠️ This is different than how Add Variation works in a Content Split Test setup.

  3. When all variations have been added, click Continue to move to the next step.

💡 Helpful Tips for Composing a Split Test

  • Need to remove a variation?
    Click Delete within the Actions icon to remove a variation.

  • Need to send a test message for any of the variations?
    Simply click Send Test within the Composition panel of the variation you would like to test. ⚠️ A test list must be set up to perform this function. Learn more about how to set up a list here.


Split Test Settings

The next step is to choose the testing criteria and the duration of the test.

  1. Enter a description of the Split Test (100 characters max) - Use a description that will best summarize the type of test you will be running.

  2. Select the Winning Metric - Choose either Click-Through-Rate or Conversion Rate. The type of content being tested will help to determine the winning metric, reference the examples below when setting up the split test:

    • Click-Through-Rate (Percentage of customers who click on a link) -
      Images variations, Images vs Text, Calls-to-Action, Headline, Slang or
      Abbreviations vs Spelled-out Words or, Message Content-Length.

    • Conversion Rate (Number of conversions divided by the number of site visits) - Incentive vs No Incentive, Offering Different Incentives, Word Order, Headlines, Calls-to-Action, or Attachment vs No Attachment.

  3. Configure the Variation Distribution - Choose to maintain the same distribution ratio for each variation (default) or send the winning message to a remainder group.

    Ratio distribution option:

    1. This default setting is automatically calculated based on the number of variations in the split test.

      • 2 Variations (minimum requirement) - 50 / 50

      • 3 Variations - 33 / 33 / 33

      • 4 Variations - 25 / 25 / 25 / 25

      • 5 Variations - 20 / 20 / 20 / 20 / 20

    2. Set the duration by entering the number of Days (range 1-7) or Hours (max 168) to run the test.

    Winning message to the remaining group option:

    1. Click the box next to the option to enable.

    2. Configure the size of the remainder group by sliding the bar in that row.

      💡 The calculations for the variations will be redistributed equally based on the ratios listed above.

    3. Set the duration - Choose by ending the test after a number of days/hours (default) or by setting a specific value based on the Winning Metric chosen in Step 2.

      • To specify a metric, click the box next to the option to enable.

      • Set the number of days/hours

      • Set the winning metric value

  4. Click Continue to move on to the next step.


Review & Send

On the Review & Send page confirm the various settings, make any last-minute edits and schedule the test.

Editing the Split Test

Click on the Edit button to update any of the configuration options or message variations.

Schedule the Send

  • For a Content Split Test

    • Choose the send date and send time (EST).

    • You can select 'Time Zone Optimized' to send at the specified time in a contact's time zone. You will be able to see how many contacts received messages in each time zone when viewing reporting on the individual message. Messages must be scheduled at least two hours in advance when using this feature.

  • For a Send Time Split Test

    • Simply select different send times (EST) for each variation.

      ⚠️ The Time Zone Optimized feature is either on or off for all variations.

Confirm and schedule

The last step to complete the split test setup is to confirm all the settings, variations, and scheduled date/time, to do so click Confirm and the bottom of the page.



Reporting

The SMS Split Test Dashboard displays important information ranging from the test status, the winning variation, revenue earned, and more about each split test created on your sender code. Access this dashboard by navigating to Messages > SMS SMS Split Test.

Split Test Dashboard

  • New Split Test: Click to create a new test.

  • Search: Use the name to find a split test on the dashboard.

  • Name: The name of the split test. Click the name to view the report dashboard for the individual split test.

  • Send Date: The start date and time of the split test.

  • End Date: The end date and time of the split test.

  • List Name: The name of the list the split test was sent to.

  • Status: Displays the current status of the test.

    • Draft - test is still being configured.

    • In Progress - test is in progress.

    • Scheduled - test is scheduled to begin in the future.

    • Completed - test is complete with results available.

  • Actions: Click the icon and select an option. These vary based on the status of the test:

Actions

Definition

Available Status

Edit

Allows you to make changes to any part of the split test.

Draft

Duplicate

Allows you to make a copy of the split test.

Draft, Scheduled, In Progress, or Completed

Delete

Allows you to remove a split test.

Draft or Scheduled

View Report

Allows you to access the individual split test report dashboard.

In Progress or Completed


Individual Split Test Reporting Dashboard

Test Details

  • Split Test Name (required): The name of the split test.

  • Export: Click to export the split test results (Excel file format).

  • Split Test Description (optional): The name used to describe the content/goal of the Split Test.

  • Number of Variations: Shows number of variations in the test (minimum 2, max 5).

  • Test Type: Indicates the type of test - Content.

  • Campaign Name: The name given to identify the split test in your Google Analytics dashboard.

  • List: The name of the list the split test was sent to.

  • Filters Applied (optional): Displays any filters applied to this test.

  • Status/Date: The current status of the test and duration of the test (date/time).

  • Winner: The name of the winning variation.

  • Winning Metric: The metric that determined the winning variation.

Message Variation Details

Click the tab associated with each variation to see the message details.

  • Message Type: Indicates the type of message - SMS or MMS.

  • Tracking Compliance: Indicates if the compliance messaging was pasted into the message.

  • Revenue:

    • Total Revenue: The total amount of revenue generated by the message.

    • Average Order Value (AOV): The average order value of orders placed as a result of the message.

  • Conversions:

    • Count: The total number of conversions made from the message.

    • Rate: The percentage is calculated by the total number of messages sent divided by the total number of conversions made.

  • Click-Through:

    • Count: The total number of links clicked.

    • Rate: The percentage is calculated by the total number of messages sent divided by the total number of links clicked.

Split Test Messages

This section displays the metrics for every variation of the split test so you can compare the performance of each message.

  • Name: The variation name applied to the test (minimum 2, max 5).

  • Winning Variation: The winning variation is marked with the status of 'Winner'.

  • Type: Indicates the type of message - SMS or MMS.

  • Send Date: The start date and time of the split test.

  • Status: Displays the current status of the test.

    • In Progress - test is in progress.

    • Scheduled - test is scheduled to begin in the future.

    • Completed - test is complete with results available.

  • Sent: The number and percentage of messages sent of the variation.

  • Delivery Rate: Percentage of the number of messages sent of the variation compared to total messages sent.

  • Failed Rate: Percentage of the number of messages failed to be delivered of the variation compared to total messages sent.

  • Capped: The number of messages not sent to contacts because they had met the frequency capping threshold.

  • Click-Through: The number and the percentage of links clicked.

  • Conversions: The number and percentage of conversions made by the message.

  • Revenue: The total amount of revenue generated by the message.

  • Average Order Value (AOV): The average order value of orders placed as a result of the message.

Did this answer your question?