A holdout group is a selection of random contacts who are withheld from receiving an email, or marketing treatment. The purpose of withholding from a treatment is to compare performance of the holdout group to those who were subjected to the treatment or email.

Interested in learning more about using holdout groups in your email marketing program? Contact your Account Manager for more information!

HOLDOUT GROUP SETUP

  1. Confirm you are on the Master List.
  2. Navigate from the home menu to Authoring.
  3. Go to Automated Testing then Holdout Groups
  4. Click New Holdout Group.           
  • TIP: All active or scheduled holdout groups must be completed or stopped to create a new holdout group.  

NEW HOLDOUT TESTS 

SIZE

The size of the holdout group determines the percentage of contacts that will be part of the test. A treatment group of the same size will be randomly selected for comparison, however the number may become smaller if filtering is used in the messaging.       

  • TIP: You can choose from between 1% to 50% using the slider, or up to two (2) decimal places by typing in the number in the text field. 

DURATION

The duration of the holdout group starts at midnight on the Start Date. The Test Duration can be set to last up to ninety (90) days.  

  1. Select a Start Date.
  2. Enter the Test Duration
  3. Click Save. 

MAINTAINING HOLDOUT GROUPS 

STATUS

A holdout group is either Scheduled or Active, and you can Stop Test at any time. After the duration expires, the holdout group refreshes according to the pre-defined logic in the test. The expired holdout groups are listed on the Holdout Groups management page that lists the following for each completed holdout group:   

  • Name
  • List Name 
  • Start Date
  • Duration
  • End Date 
  • Holdout Size

EXPORT

Holdout Groups allows users to export a list of records included in a holdout group, whether the holdout group is active or expired. Scheduled holdout groups are not available for export. Upon export, two files are produced: one containing the records in the holdout group, and another containing those in the treated group.

  1. Click the three dots in the Actions column. (For active holdout groups, click the Export button next to the Stop Test button.)
  2. Click Export

   3. Click the Export button.  

OVERRIDE

There is an override option where users can elect to include the holdout group contacts when authoring a new message. There may be times where you determine a marketing message is important and should be sent to all contacts. Overriding the holdout group at the message level allows contacts to receive the message.

  1. Navigate to Authoring then New Message.
  2. Scroll to the Advanced Message Settings.  
  3. Check the Holdout Group checkbox

LIST REVIEW GROUP

If specific addresses are in a holdout group but need to receive emails (e.g., internal users who need to receive all messages they would normally qualify for), they can be included in the List Review Group.  Addresses in the List Review Group will not be excluded from a send if they are included in a holdout group.

The List Review Group can be found navigating from the home menu to Manage > Lists > List Settings.  Then navigate to Notification Settings.

Interested in learning more about Holdout Groups? Check out our video for more information:

Did this answer your question?