Test Set-Up & Analysis

Predicting sample size

Understanding the required sample size will help you determine how long the test should run to achieve statistical significance.

Optimizely has a useful sample size calculator here: https://www.optimizely.com/uk/sample-size-calculator/

This calculator helps you determine the necessary sample size per variation to accurately measure the expected change in your conversion rate.

Consider the minimum detectable effect (MDE) you aim to measure. It should be significant enough to address any "so what?" questions from stakeholders.

For example, if the test aims to increase conversions by 3%, this could generate an additional €6 million in yearly revenue.

Letting the test run

Let the test run its course. This often requires patience and resisting the urge to stop the test due to unexpected fluctuations. You can monitor your currently active A/B-test directly in Insights under A/B-test reports / Live experiments in the relevant full report.

In the highlighted section of the graph below, you'll notice that all variations behave unpredictably at the start of the test. Although it may be tempting to end the test early, the graph shows that variations tend to stabilize over time.

In the box highlighted in the graph below, you can see that one variation suddenly peaks and may even reach a significant change. You must take into consideration external factors which could have influenced this change and continue to let the test run for the full duration.

As shown in the graphs above, performance fluctuates in the first few days but stabilizes over time.

Statistical significance

A good statistical significance calculator tool to use: https://abtestguide.com/calc/

 

Quantitative & Qualitative Data

Analyze both quantitative and qualitative data to understand not just what is happening, but why.

Why did these changes occur? What do we predict will happen if we modify X next? This can be crucial in forming follow-up testing hypotheses.

To effectively use both quantitative and qualitative data use a convergent parallel design:

1. Collect quantitative and qualitative data simultaneously.
2. Independently analyze the results.
3. Compare and contrasts the results to look for patterns or contradictions.
4. Converge and compile the findings to generate insights.

We mix quantitative and qualitative research methods to not just understand what is happening but why it is happening.

A/B testing tips

Do:

• Analyze test results thoroughly – Understand why the winning variation succeeded and why the losing variant did not.
• Leverage additional tools – Use platforms like Google Analytics to segment data and evaluate how each variation performed across different groups.
• Look for segment-specific trends – The losing variation may have performed well for a particular customer segment.
• Continuously generate new hypotheses – Always aim to maximize insights by testing new ideas.
• Tag non-time-based cohorts – This ensures the full picture can be analyzed effectively.
• Use Analytics annotations – Keep track of anomalies and key events to make data interpretation easier.

Don’t:

• Make changes mid-test – Avoid making changes mid-test, as this prevents accurate assessment of each variation’s impact.
• End the test prematurely – Don’t end the test too early—allow it to run its full course, even if early results are unstable.
• Overlap tests – Be mindful of conflicts with other tests that could be running simultaneously.
• Allow users to switch between test groups – Prevent users from switching between test groups to maintain test integrity and avoid overlap.