The ability to perform A/B split tests within your SMS program will provide valuable insight into the type of content that drives engagement or increases conversion rates to your entire SMS list or simply apply a one-time filter to refine the number of contacts who will be a part of the test.

There are numerous strategies you can employ in a content-type split test, consider one of the following options:

  • Attachment vs No attachment (SMS vs MMS)

  • Word Order

  • Personalization vs No Personalization

  • Calls-to-Action

  • Message Content-Length

  • Subject Lines

  • Subject Line vs No Subject Line

  • Images: People vs Products

  • Images: with Copy vs Image-only

  • Images: Static vs Animated GIF

  • Incentive vs No Incentive

  • Offer Different Incentives

  • Slang or Abbreviations vs Spelled-out Words

  • Capitalization vs Lowercase

  • Hard Returns vs One block of copy

⚠️ Cadence/Schedule Time of Day Test is not available to be tested.


Content Split Test Setup

To create a content split test, follow the steps below:

  1. From the SMS Home Dashboard click the New Campaign button.

  2. Select the Split Test Content (A/B) card.

  3. Enter a name for the split test.
    💡 Click Save at the bottom of the control panel.

  4. Next configure the Setup, Compose, Split Test Settings, and Review & Send components of a split test.

Setup

The setup consists of selecting the list, the option to add a one-time filter, and apply both Listrak and Google Analytics to the split test. Click here for instructions on completing the Setup configurations.

⚠️ Split test messages can not be used for data acquisition.

Click Continue to move to the next step.

💡 Click the Step name in the left sidebar to configure it at any time while in the split test builder. The split test does not need to be built in a specific order.


Compose

Variation Setup

Once the strategy has been determined, the next step is to set up the number of variations and apply the message content to each by following these steps:

  1. Click Add Variation - The tool allows you to test 2, 3, 4, or 5 different variations of content. The distribution of the test will be weighted equally regardless of the number of variations you choose.

2. Within each variation, click EDIT to add the message content. Follow the same
steps when creating a Broadcast Message. Clicking EDIT will also allow you to
to make any additional changes to the message content.
⚠️ Subject Lines are not available in the Split Test tool.

3. Click Continue to move to the next step.

💡 Helpful Tips for Composing a Split Test

  • Want to test similar content?
    Click Duplicate within the Actions icon to duplicate a variation. Then click EDIT to make small changes - like adding a link, attaching a file, or adding personalization. Click Apply to save!

  • Need to remove a variation?
    Click Delete within the Actions icon to remove a variation.

  • Need to send a test message for any of the variations?
    This option is not available in the split test builder. However, you can recreate a specific variation in the Broadcast Message builder and send yourself a test!


Split Test Settings

The next step is to choose the testing criteria and the duration of the test.

  1. Enter a description of the Split Test (100 characters max) - Use a description that will best summarize the type of test you will be running.

  2. Select the Winning Metric - Choose either Click-Through-Rate or Conversion Rate. The type of content being tested will help to determine the winning metric, reference the examples below when setting up the split test:
    ▪️ Conversion Rate (Percentage of customers who click on a link):
    Incentive vs No Incentive, Offering Different Incentives, Word Order, Headlines,
    Calls-to-Action, or Attachment vs No Attachment.
    ▪️ Click-Through-Rate (Number of conversions divided by the number of site
    visits):
    Images variations, Images vs Text, Calls-to-Action, Headline, Slang or
    Abbreviations vs Spelled-out Words or, Message Content-Length.

  3. The Variation Distribution will automatically display the estimated subscriber count and weighted distribution for each variation.

Number of variations

Weight %

2 Variations (minimum requirement)

50 / 50

3 Variations

33 / 33 / 33

4 Variations

25 / 25 / 25 / 25

5 Variations

20 / 20 / 20 / 20 / 20

4. Enter the number of Days (range 1-7) or Hours (max 168) to run the test.

5. Click Continue to move onto the next step.


Review & Send

On the Review & Send page you can confirm the various settings of your split test. If need, click on the Edit button to edit any of the configuration options.

Once you have confirmed and Saved the settings you are ready to Schedule a One-Time Send!



Reporting

The SMS Split Test Dashboard displays important information ranging from the test status, the winning variation, revenue earned, and more about each split test created on your short code. Access this Dashboard by navigating from the home menu to Campaigns > Split Tests.

Split Test Dashboard

  • New Split Test: Click to create a new test.

  • Name: The name of the split test. Click the name to view the report dashboard for the individual split test.

  • Send Date: The start date and time of the split test.

  • End Date: The end date and time of the split test.

  • List Name: The name of the list the split test was sent to.

  • Status: Displays the current status of the test.

    • Draft - test is still being configured.

    • In Progress - test is in progress.

    • Scheduled - test is scheduled to begin in the future.

    • Completed - test is complete with results available.

  • Actions: Click the icon and select an option. These vary based on the status of the test:

Actions

Definition

Available for X Status

Edit

Allows you to make changes to any part of the split test.

Draft

Duplicate

Allows you to make a copy of the split test.

Scheduled, In Progress, or Completed

Delete

Allows you to remove a split test.

Draft or Scheduled

View Report

Allows you to access the individual split test report dashboard.

In Progress or Completed


Individual Split Test Reporting Dashboard

Test Details

  • Split Test Name (required): The name of the split test.

  • Split Test Description (optional): The name used to describe the content/goal of the Split Test.

  • Number of Variations: Shows number of variations in the test (minimum 2, max 5).

  • Test Type: Indicates the type of test - Content.

  • Campaign Name: The name given to identify the split test in your Google Analytics dashboard.

  • List: The name of the list the split test was sent to.

  • Filters Applied (optional): Displayed any filters applied to this test.

  • Status/Date: The current status of the test and duration of the test (date/time).

  • Winner: The name of the winning variation.

  • Winning Metric: The metric that determined the winning variation.

Message Variation Details

Click the tab associated with each variation to see the message details.

  • Message Type: Indicates the type of message - SMS or MMS.

  • Tracking Compliance: Indicates if the compliance messaging was pasted into the message.

  • Revenue:

    • Total Revenue: The total amount of revenue generated by the message.

    • Average Order Value (AOV): The average order value of orders placed as a result of the message.

  • Conversions:

    • Count: The total number of conversions made from the message.

    • Rate: The percentage is calculated by the total number of messages sent divided by the total number of conversions made.

  • Click-Through:

    • Count: The total number of links clicked.

    • Rate: The percentage is calculated by the total number of messages sent divided by the total number of links click.

Split Test Messages

This section displays the metrics for every variation of the split test so you can compare the performance of each message.

  • Name: The variation name applied to the test (minimum 2, max 5).

  • Winning Variation: The winning variation is marked with the status of 'Winner'.

  • Type: Indicates the type of message - SMS or MMS.

  • Send Date: The start date and time of the split test.

  • Status: Displays the current status of the test.

    • In Progress - test is in progress.

    • Scheduled - test is scheduled to begin in the future.

    • Completed - test is complete with results available.

  • Sent: The number and percentage of messages sent of the variation.

  • Delivery Rate: Percentage of the number of messages sent of the variation compared to total messages sent.

  • Failed Rate: Percentage of the number of messages failed to be delivered of the variation compared to total messages sent.

  • Click-Through: The number and the percentage of links clicked.

  • Conversions: The number and percentage of conversions made by the message.

  • Revenue: The total amount of revenue generated by the message.

  • Average Order Value (AOV): The average order value of orders placed as a result of the message.

Did this answer your question?