This document explains the metrics used by the A/B test reports available in Insights, where these values come from, and how to interpret them.

The metrics displayed for live and archived experiments are identical. Examples provided in this document refer to a live experiment for clarity.

For a more general introduction on how to use Insights visit Inside Insights: A user guide.

Warning: Numbers in the A/B test reports section are calculated using data from the entire FAS environment selected, not from the data scope defined using Global Filters (unless clearly indicated otherwise).

A/B test report summary page

The A/B test report summary page shows a list of all live or archived experiments from the selected FAS environment.

For each experiment, you can see:

The experiment title under and next to which you can find the type of experiment ( Search , Campaigns, Rankings , Result modification)

The sample size indicating the total number of impressions for all variants (displayed in bold) as well as average impressions for each variant.

Note: A variant in an A/B test is defined as a specific version of the search, campaign, ranking or result modification experience that differs from the baseline and that you want to test. An A/B test has at least two variants, one of which representing the baseline. A/B tests where more than two scenarios are tested are called multi-variant or A/B/N test.

The winning variant indicating if a winner has been found, displaying its name and the primary KPI used to select it, i.e view conversions.

Note: In the summary only the primary KPI is taken into account when determining whether there is a winner among the tested variants. To establish whether a significant winner was determined for a secondary KPI you have to consider the full report.

The confidence with which the winner for the primary KPI has been determined and whether it is significant; you can read more about significance and confidence here.

The start date representing the day on which the test was published as well as the number of days it has been running; note that if you published a test, paused it, and then published it again, this still shows the date on which the test was first published.

Summary table

Expanding the information for an experiment using opens the summary table . The summary table shows:

The name of the variants tested where B. indicates the baseline variant.

The impressions per variant indicating how often each variant tested has been viewed.

The primary and secondary KPIs for each variant:

• View conversions (primary KPI)
• Revenue per unique impression

Note: Revenue per unique impression is the total revenue generated during the experiment divided by the number of unique impressions for a variant. A unique impression is counted only the first time a Fredhopper-triggered entity is loaded.
Because unique impressions reflect the number of distinct user journeys in which a variant was first shown, revenue per unique impression helps indicate how effectively a variant generates revenue relative to the number of unique opportunities it had to influence a shopper.

• Basket conversions
• Purchase conversions

The lift for each KPI measuring the percentage change of a variant compared to the baseline for a given KPI, showing how much better or worse the variant performs; a positive lift means the variant performs better, while a negative lift indicates worse performance.

Full report

You can reach the full report for an experiment by clicking . The full report is divided into the following five sections:

General information: Here you can find basic information such as the status and type of test, total number of views, number of days the test has been running, and when it was last modified.

Summary table: This table is almost identical to the summary table on the summary page. It additionally shows the confidence for secondary KPIs and indicates whether there has been a winner. Winners are marked in bold green.

KPI trend graph: This graph shows how each variant performs over time for the selected metric. It plots the daily values on a timeline so you can visually compare trends between variants and see how performance evolves throughout the test. For more information on the available metrics, see Metrics: Definitions and Examples.

Detailed metrics table: This table shows the selected metrics for the tested variants.

Labels and notes section: Here you can add labels to categorize your A/B tests and leave notes for yourself and others.

Warning: The KPI trend graph and the detailed metrics table show data based on the data scope selected in Global Filters.