Reading results
What the experiment report shows, how lift and confidence are calculated, and how much data you need before trusting a result.
The Experiment card on a push or in-app report shows one row per group, a one-sentence summary and, for a push with a test share, the button to send the winner. Journey reports show the same figures for the whole journey and for each split.
The table
| Column | Meaning |
|---|---|
| Group | Variant A, B or C, Control group, and Project control group when your project holds people back. |
| People | People exposed in the date range. See Who counts as exposed. |
| Opened / Clicked | People who opened the push or clicked the in-app message, and the rate. Empty for control groups. |
| Converted | People who reached the goal within the window, and the rate. Empty without a goal. |
| Revenue | The group's net store revenue within the window. |
| Per person | Revenue divided by People. |
The range starts on the campaign's start day and covers up to 92 days. Numbers refresh about once a minute.
Variant against variant
Each variant after A gets a confidence figure: how sure Grovs is that it beats A. It compares conversion rates with a goal, open or click rates without.
It appears once both groups have at least 100 people. Until then, the report says there aren't enough people yet.
Message against control
With a goal and a control group, the report pools all variants and compares them with the campaign's control group:
| Figure | How it's worked out |
|---|---|
| Lift | Variants' conversion rate divided by the control's, minus 1. A 20% lift means people who got the message converted 1.2 times as often. Empty when the control converted nobody. |
| Added conversions | Variants' conversions minus what they would have had at the control's rate. |
| Confidence | How sure Grovs is that the message beats doing nothing. |
| More people needed | Under 95% confidence: roughly how many more people each group needs for a difference this size to reach 95%. |
The project control group is shown for reference only and isn't used in these figures.
Example summaries:
- "Variants convert 18% better than the control: about +240 conversions (97% confidence)."
- "Not enough people yet for a clear answer: about 3,400 more per group."
A result is clear at 95% confidence. When the variants do worse than the control, the summary says so, with the confidence that they are worse.
With a control group, the report also shows how much more or less revenue the variants earned than the control, in total and per person. Revenue has no confidence figure.
How confidence is calculated
Grovs uses a one-sided two-proportion z-test:
- Pool both groups to get the expected rate if they performed the same.
- Measure the gap between the observed rates, in units of the spread expected from chance at that pooled rate and those group sizes.
- Convert that distance into a confidence from 0 to 100%.
At 95% confidence, if both groups truly converted at the same rate, a gap this large in this direction would appear by chance about 5% of the time.
"More people needed" uses the standard two-rate sample-size formula at 95% one-sided confidence and 80% power, with the rates seen so far. It changes as the rates move.
Reading it well
- Decide up front. Pick the goal and the run length before you look. Checking hourly and stopping at the first 95% overstates certainty.
- Watch multiple variants. Each variant is compared with A separately. With both B and C, the chance one looks better by luck is higher than either figure suggests.
- Small differences need many people. A 10% conversion rate moving to 11% needs roughly 12,000 people per group. A move to 15% needs about 540.
- The automatic winner isn't a significance test. It sends the best rate when the timer ends, as long as every variant reached 100 people. For decisions that matter, pick the winner yourself once the report shows a clear result.
- Trigger-based in-app messages count only people who saw the message, so lift can look better than it is. See Who counts as exposed.