Experiments
Compare message variants, hold back a control group, and find out whether a push, in-app message or journey changed what people do.
An experiment tells you which version of a message works better, and whether sending it changed anything. Run one on a push campaign, an in-app message or a journey. Set it up in the message's Experiment step and read it on the report.
Combine any of these parts:
| Part | What it does | Push | In-app | Journey |
|---|---|---|---|---|
| Goal | What counts as a conversion, such as an event or a purchase within a time window. | Yes | Yes | Yes |
| Control group | A share of the audience that doesn't get the message, for comparison. | Yes | Yes | Yes |
| Variants | Up to three versions, A, B and C, split evenly. | Title and body | Content | Percentage splits |
| Test share and winner | Send the variants to part of the audience first, then the winner to the rest. | Yes | No | No |
Every experiment report shows store revenue, with or without a goal.
How people are split
Each person gets a fixed number from 0 to 99 per campaign, derived from the campaign and the person. They always land in the same group for that campaign. Grovs counts people by visitor, and a visitor is one device, so someone with two phones can land in a different group on each.
- The lowest numbers form the control group. With a 10% control, 0 to 9 are held back.
- The rest split evenly between variants. With a 10% control and variants A and B, 45% get A and 45% get B.
Because the number depends on the campaign, being in one campaign's control group doesn't keep someone from the next.
Devices that can't receive the message are never held back. A push device with no token counts as No channel, not control.
Control group
In the Experiment step, choose No one held back, or 5, 10, 20 or 50% held back. The API accepts any whole number from 1 to 50. The control group is fixed once the campaign is out.
The control group gets nothing from this campaign. Comparing them with people who got the message shows what the message changed. That comparison needs a goal.
Project control group
Engage › Engagement Settings › Control group holds back 0 to 20% of everyone in the project from all in-app messages, journeys and dashboard push campaigns. It uses a project-based number, so it is the same people across every campaign. Server pushes aren't held back.
Every experiment report shows these people as Project control group: the effect of your messaging as a whole, next to one campaign's.
Variants
- Push: in the Message step, add variant B, and optionally C, each with its own title and body. Image, link and data are shared. Each variant must fit Apple's 4 KB limit.
- In-app: each variant is a full copy of the content for the same template. After launch you can edit a variant's content, but not add or remove variants.
- Journeys: a split step sends people down two or three paths by percentage. See Journeys.
Without a goal, variants are compared on opens (push) or clicks (in-app). With a goal, on conversions.
Test share and winner (push only)
Send the variants to part of the audience first, then the best one to everyone else.
Under Test share, pick 10, 20, 30 or 50% of the audience. Everyone at once sends a variant to every person straight away. The rest of the audience shows as Waiting for the winner.
Under Pick the winner, choose Myself, from the report, or Automatically after 30 minutes, 1 hour, 2, 6, 12 or 24 hours.
Under Judge by, choose Opens or Conversions. Conversions need a goal.
Automatic. When the timer ends, the variant with the best rate goes to the rest. Ties go to the earlier variant. If any variant has reached fewer than 100 people, A goes. The timer doesn't wait for a clear result.
Manual. On the report, click Send variant B to the rest (or A, or C), even before an automatic timer fires. It can't be undone.
A test share can't be combined with local-time delivery. People still waiting 30 days after the send are released and don't get the push.
Rolling out an in-app winner
On an in-app experiment report, Use for everyone next to a variant creates a new draft with that variant's content, and can end the current campaign. Launch the draft to show the winner to everyone.
Who counts as exposed
The report compares people exposed in each group:
- Push: everyone handed a variant when the audience is frozen, delivered or not. The control group is counted at the same moment. Failed pushes stay in, so no variant is flattered by dropping them.
- In-app: people who saw the message. The control group counts people who would have been shown it when their app checked for messages.
For trigger-based in-app messages, viewers reached the trigger, so they are often more active than the control group overall. Read the lift as an upper bound.
Next: Goals and Reading results.