Everything to know about

A/B Testing

by letthedataconfess 

A/B testing is a basic randomized control experiment. It is a way to compare the two versions of a variable to find out which performs better in a controlled environment. It is a hypothetical testing methodology for making decisions that estimate population parameters based on sample statistics.
For example: If you own a company and want to increase the sales of your product. You may divide the products into two parts – A and B. Here A will remain unchanged while you make significant changes in B’s packaging. Now, on the basis of the response from customer groups who used A and B respectively, you try to decide which is performing better.

Null Hypothesis

There is no difference between the control and variant groups.
Alternative Hypothesis: There is a difference between the control and variant groups, i.e. your A/B test is true.

If the experiment turned out to be statistically significant, we reject the null hypothesis and accept the alternate hypothesis. To reject the null hypothesis, the p-value should be less than the confidence level ( significance level) i.e. 0.05

What is Control Group and Test Group?

We need to divide the group of people participating in the test into two groups.

Control Group is the one which receives the original version of the product.
Test Group is the one which receives the product with some significant changes that the company is planning to do.

How are groups formed?

  1. The control and the test group are formed using random sampling from the population. Random sampling is important in hypothesis testing because it eliminates sampling bias, and it’s important to eliminate bias because you want the results of your A/B test to be representative of the entire population rather than the sample itself.
  2.  We must also determine the minimum sample size for our A/B test before conducting it so that we can eliminate under coverage bias, which is the bias from sampling too few observations.

How is the test conducted?

To prove the statistical significance of our experiment we can use a two-sample T-test which will test whether the difference between your control version and the test version is due to some error or random chance or not.
The two–sample t–test is one of the most commonly used hypothesis tests. It is applied to compare whether the average difference between the two groups.

When is A/B testing used?

  1. A/B testing is used by both startups and giant businesses to increase the traffic and conversion.
  2.  A/B testing is majorly used when testing incremental changes, such as UX changes, new features, ranking, and page load times.