A/B testing is a basic randomized control experiment. It is a way to compare the two versions of a variable to find out which performs better in a controlled environment. It is a hypothetical testing methodology for making decisions that estimate population parameters based on sample statistics.
For example: If you own a company and want to increase the sales of your product. You may divide the products into two parts – A and B. Here A will remain unchanged while you make significant changes in B’s packaging. Now, on the basis of the response from customer groups who used A and B respectively, you try to decide which is performing better.
There is no difference between the control and variant groups.
Alternative Hypothesis: There is a difference between the control and variant groups, i.e. your A/B test is true.
If the experiment turned out to be statistically significant, we reject the null hypothesis and accept the alternate hypothesis. To reject the null hypothesis, the p-value should be less than the confidence level ( significance level) i.e. 0.05
We need to divide the group of people participating in the test into two groups.
Control Group is the one which receives the original version of the product.
Test Group is the one which receives the product with some significant changes that the company is planning to do.
To prove the statistical significance of our experiment we can use a two-sample T-test which will test whether the difference between your control version and the test version is due to some error or random chance or not.
The two–sample t–test is one of the most commonly used hypothesis tests. It is applied to compare whether the average difference between the two groups.