When assessing statistical significance, avoid the most common mistakes:
do not believe in 100% consumer acceptance of a new page, because this method only demonstrates such a probability and reduces the possibility of chance;
A/B testing does not directly show that one or another option is more popular with the user. Its result only represents the superiority of some target indicators over others;
Even if the difference between pages is obvious, remember that users did not directly choose this, they were not given two options. The analysis only showed their reaction to changes beyond their control.
Focus group testing
Source: shutterstock.com
Sometimes the results of A/B tests engineer database do not give an obvious result for the variants. In this case, it is necessary to resort to the analysis of specific segments by certain features:
demographic characteristic (gender, age, country);
division into newcomers and professionals (it is more objective to use only newcomers);
division by payers;
division by audience source and service.
Noticeable differences across segments can help shed light on the true results of your A/B testing.
To correctly divide traffic into parts, consider two main rules:
The sample size in each segment must be reasonable and convincing. The generally accepted minimum number of conversions per test variant is 250–350;
too small a sample is not suitable for A/B testing.
Download a useful document on the topic:
Checklist: How to Achieve Your Goals in Negotiations with Clients
Examples of A/B testing from world-famous companies
Netflix
This online cinema is very successful in large part because it regularly tests its ideas for changing the site.
For example, Netflix personalization is carried out in such a way that the visitor can easily navigate the platform. In addition, the user should be surrounded by thematic selections compiled based on his individual interests. The element of randomness here is reduced to a minimum.
Netflix
Source: unsplash.com
In addition to personal pages, Netflix intelligently selects the most clickable titles and thumbnails offered to the user's view.
Amazon
As a recognized leader in conversion optimization, Amazon strives to provide its customers with the best possible experience.
In the late 90s, after conducting numerous long-term tests, the company launched an option called "One-Click Order". This allowed the client to buy goods without unnecessary movements. One-time addition of your bank card and address data significantly simplified the process of making a purchase. This innovation received such a massive response that most clients refused to work with other platforms. After the success of its idea, Amazon filed a patent for this option, the license for which was soon acquired by the well-known company Apple.