AIDEM KB

Experiments

Background

Customers can target different line items to sets of non-overlapping users, allowing comparison of results to test performance. Also known as “A/B Testing” or “Randomized Control Studies”, the Experiments feature creates isolated subsets of users, and ensures these users are only exposed to one Line Item setup.

Use Cases

  • Compare Bidding Strategies: You want to test two or more different bidding methods to see which gives the best result. You can do this with identical line items for targeting but with audience splits.
  • Compare Multiple Creatives: You want to test two or more sets of creatives or landing pages to see which generates more conversions. You can do this by creating one line item per set of creatives and assigning a different test group to each.
  • Persistent Control Group: You want to set aside a control group of creatives (PSAs) to measure incremental lift. You can do this by using the X Percent Hold-out Test Plans. Assign the PSA line item to the Control Group (small portion of users), and all other line items should be assigned to the Test Group (large portion of users).

How to use the Experiments feature:

Experiments can be enabled at the Campaign level, with Test Plans being assigned at Campaign-level and Test Groups at the Line Item level:

  1. A new toggle on Campaign: Campaigns have the Experiment toggle.

    • When the toggle is set to Yes, a new Experiment tab appears on the object that allows a Test Plan to be associated with the campaign.
    • When the toggle is set to No, the experiment tab does not appear. The toggle is set to No by default.
    • If the toggle is set to Yes but no Test Plan is selected in the Experiment tab before closing the window, the toggle gets reset to No. It is recommended to set up the Experiment after you create both the Campaign and Line Item.

  2. Test Plans and Test Groups: Test Plans and Test Groups are pre-configured.

    • There are 14 Test Plans available. Each test plan has a different split of users arranged into Test Groups, allowing users to pick the plan that fits their use case the best.
    • Test Plans are associated at the campaign level and Test Groups are associated at the line item level.
    • You can only assign test groups to line items whose campaigns have an assigned test plan.
Test PlanTest GroupsUser Index Range1
Five Percent Hold-out Plan (ID: 1)Control Group (5%) (ID: 1)
Test Group (95%) (ID: 2)		1-50, 51-1000
Ten Percent Hold-out Plan (ID: 2)Control Group (10%) (ID: 3)
Test Group (90%) (ID: 4)		1-100, 101-1000
Twenty Percent Hold-out Plan (ID: 3)Control Group (20%) (ID: 5)
Test Group (80%) (ID: 6)		1-200, 201-1000
Two-Way Split Plan (ID: 4)Group A (50%) (ID: 7)
Group B (50%) (ID: 8)		1-500, 501-1000
Three-Way Split Plan (ID: 5)Group A (33.3%) (ID: 9)
Group B (33.3%) (ID: 10)
Group C (33.4%) (ID: 11)		1-333, 334-666, 667-1000
Ten-Way Split Plan (ID: 6)Group A (10%) (ID: 12)
Group B (10%) (ID: 13)
Group C (10%) (ID: 14)
Group D (10%) (ID: 15)
Group E (10%) (ID: 16)
Group F (10%) (ID: 17)
Group G (10%) (ID: 18)
Group H (10%) (ID: 19)
Group I (10%) (ID: 20)
Group J (10%) (ID: 21)		1-100, 101-200, ..., 801-900, 901-1000
Three-Way Split with Fifty Percent Hold-Out Plan (ID: 7)Control Group (50%) (ID: 22)
Group A (25%) (ID: 23)
Group B (25%) (ID: 24)		1-500, 501-750, 751-1000
Twenty Five Percent Hold-out Plan (ID: 8)Control Group (25%) (ID: 25)
Test Group (75%) (ID: 26)		1-250, 251-1000
One Percent Hold-Out Plan (ID: 9)Control Group (1%) (ID: 27)
Test Group (99%) (ID: 28)		1-10, 11-1000
Three-Way Split with Ten Percent Hold-Out Plan (ID: 10)Control Group (10%) (ID: 29)
Group A (45%) (ID: 30)
Group B (45%) (ID: 31)		1-100, 101- 550, 551-1000
Three-Way Split with Eighty Percent Hold-Out Plan (ID: 11)Control Group (80%) (ID: 32)
Group A (10%) (ID: 33)
Group B (10%) (ID: 34)		1-800, 801-900, 901-1000
Three-Way Split with Ninety Percent Hold-Out Plan (ID: 12)Control Group (90%) (ID: 35)
Group A (5%) (ID: 36)
Group B (5%) (ID: 37)		1-900, 901-950, 951-1000
Four-Way Split Plan (ID: 13)Group A (25%) (ID: 38)
Group B (25%) (ID: 39)
Group C (25%) (ID: 40)
Group D (25%) (ID: 41)		1-250, 251-500, 501-750, 751-1000
Four-Way Split with Ten Percent Hold-Out Plan (ID: 14)Control Group (10%) (ID:42)
Group A (30%) (ID: 43)
Group B (30%) (ID: 44)
Group C (30%) (ID: 45)		1-100, 101-400, 401-700, 701-1000
Thirty Percent Hold-out Plan (ID: 15)Control Group (30%) (ID: 46)
Test Group (70%) (ID: 47)		1-300, 301-1000
1

User Index Range: each Test Group is assigned a User Index Range, which indicates the size of a Test Group and represents how the overall Experiment audience has been split. The index value of a user displays which Test Group (or index range) they fall into, and therefore which messaging/strategies they have been exposed to. The User Index value is also made available in the win-logs and allows further analysis of test results.

  1. Test Plan and Test Group Association:

    • From the Campaign object, you can associate Test Plans to a Campaign by selecting the Experiments Tab and selecting a Test Plan from the dropdown.
    • Once you select a Test Plan, select a Test Group for each of the line items within the campaign.
      • All line items within the Campaign must have an assigned test group. If not, then you cannot save the associations.
      • Multiple line items can be assigned to the same test group.
      • You do not need to select all test groups. 

* Once a Test Plan has been selected, all newly created line items show an Experiments section in their Overview section

Reporting

When using this feature, our reporting products will provide insights through the following:

Conversion Report and Performance Report include the following fields:

  • Test Group ID
  • Test Group Name
  • Test Plan ID
  • Test Plan Name

FAQS

Q: How did you come up with the uniformly sized user groups?

Based on our research, these are the most prevalent use cases for running experiments.

Why can't I apply tests to creatives instead of Line Items? It is a lot of extra work to create multiple Line Items.

When we were evaluating the different options, one important criterion is which implementation provides you with the most flexibility and allows the most use cases. Applying test groups on the line item level is the best option so far.

What should we use for Experiments using IP: IPv6 or IPv4? What happens if IP gets obfuscated?

IPv6 takes precedence when both types of IPs are present. If bid request has obfuscated IPs, then the bid request is not eligible for line items using IP as an experiment type.

What happens if a bid request does not contain the ID required for the experiment?

If an ID is not available on the bid request, the bid request is not eligible for campaign and line items running experiments.

← Presets
Overview →