A/B Testing: From Sampling to Interpretation

Hi, I’m Elia, Business Intelligence Manager at Buzzvil, a mobile first-screen media platform. Like many companies working in mobile, we improve our products through countless A/B tests. In this post, I’d like to share practical observations about questions that often arise when running them.

The idea began with a question from Jim, the PM sitting next to me:

“How do you decide the sample size for an A/B test?”

It comes up constantly at startup and data conferences, so I thought organizing my thoughts might help others. I’ll cover several related ideas in two parts: sampling and interpretation.

1. Sampling

The statistical technique commonly used in A/B testing is a t-test, which estimates whether the means of groups A and B differ. You may wonder why we need a test when both groups’ values are directly observable; I’ll return to that in the interpretation section.

For the 5% alpha error rate and 95% testing power commonly used in academia, a sample of around 120 is generally enough in the approach described here. Compared with sample sizes used in industry, this is not a large number, so it is usually not the main concern in mobile testing. Conversely, samples below 120 can create considerable difficulties.

Once sample size is sufficient, the next questions are where the real difficulties—and much of the confusion in A/B testing—begin.

Have samples been assigned randomly?

First, samples need to be sufficiently random. Random sampling matters because we want to isolate the effect of the stimulus being tested.

Consider click-through rate, one of the most common KPIs in mobile experiments. Suppose we assign 500 users to each group, show green text to A and blue text to B, then choose whichever color produces a higher CTR. Only with random assignment can text color be the sole systematic difference between the groups, allowing us to observe its effect.

What if group A contains people who especially like green? What if it contains users with naturally high CTR regardless of color? In either case, even a result of “A = 3%, B = 1%” would not justify concluding that green causes a higher CTR than blue.

How can we assign users randomly? One common approach is to order users by registration time, assign integers from 1 to N, and use the remainder from a modulo function to form groups. For two groups, mod(X, 2) creates groups 0 and 1. Since user characteristics are unlikely to differ systematically according to odd or even registration order, this removes most noise. In practical terms, it gets us more than 90% of the way toward random sampling. The remaining 10% below is a lesson from experience and may depend on the service.

A sample is—and must be—a miniature of the population

A sample must reflect the population. The previous section concerned randomness; this one addresses the more fundamental need to represent the right population. Otherwise, results can be disappointing.

Here is an example from Buzzvil, with illustrative numbers. Sometimes we test only newly registered users, or NRU, to avoid prior experience affecting their responses. Returning to text color, we once restricted a test to new users to observe their reaction to blue versus green without preconceptions. Existing users were accustomed to green and might react more strongly to a change, so we treated that experience as “noise” to remove.

The test showed green at 3% CTR and blue at 3.6%—a promising 20% improvement. We rolled blue out to everyone, only to see CTR rise from 3% to 3.15%, an improvement of just 5%.

Why? Several factors could be involved, but the main issue was that our sample did not represent our population. Relative to all users, NRU is a specific subgroup. New users have no accumulated experience of the service and no prior expectations about text color. That absence of expectations was itself the problem.

The sample looked clean for testing purposes, but our goal was to improve CTR for all users, not just new ones. Existing users had learned different things through their varying experience and responded differently to the new color. If we wanted to improve population-wide CTR, we should have sampled from that population. If learned behavior worried us, assigning users in registration order across groups could distribute that experience evenly and let its effects balance out.

Sampling should be random to balance noise, but never forget the larger requirement: it must faithfully represent the population whose KPI you want to improve.

2. Interpretation

Now we’ve sampled and exposed each group to its stimulus. How should we interpret their responses?

Suppose we ran the green-versus-blue experiment for two weeks, with 500 randomly assigned users in each group representing the population. A recorded 3% CTR and B recorded 4%. B looks better—but can we trust that result?

Understand what the t-test is assessing

What does it mean to ask whether B’s 4% is reliably higher than A’s 3%?

First, A’s 3% is not an immutable value. It varies across users and external conditions such as time or weather. That average combines users whose typical CTR is 10%, users whose CTR is 0.1%, people seeing the text while sleepy at 3 a.m., and people seeing it at 10 p.m. during prime time.

Repeat the two-week experiment with the same users and nominally identical conditions, and the average might be 3.1% or 2.7%. We simply observed 3% this time. The same applies to B: a future test might produce 3.5% or 4.2%.

So asking whether B’s 4% is reliably higher than A’s 3% is really asking whether the observed one-percentage-point difference will recur in another experiment—and persist when applied to everyone. In statistical terms, we are asking whether the observed 3% and 4% come from the same underlying distribution or different ones.

If they come from the same distribution, repeating the experiment could produce 2.5% versus 4.5%, or just as plausibly reverse the ranking with 3% versus 2%. In that case, the answer to “Can we trust it?” is no: we cannot rely on the same ordering appearing next time. For the difference to be meaningful in this explanation, the values must come from different distributions.

If the values come from different distributions as illustrated, the difference is more likely to persist in another experiment, supporting confidence that B’s CTR is higher. The difficulty is that we cannot directly see the underlying distributions. We know only that A produced 3% and B produced 4%. We must infer from that information, which is where a t-test helps.

Reading a calculated result is easy. Many online calculators are available; one option is https://abtestguide.com/calc/. It has no affiliation with Buzzvil.

The software handles the calculation, but intuitively, sample size matters. A difference observed among ten people in each group is not as convincing as the same difference among 10,000. If samples of 10,000 each produce 3% and 4%, we have stronger grounds to infer different underlying distributions.

Entering our example produced a result that was not significant. In the repeat-experiment interpretation used here, that means there is insufficient confidence that A will continue to produce a lower result than B under the same conditions. With confidence set to 95%, the intended question is whether that ordering would hold in at least 95% of indefinitely repeated experiments.

Personally, I don’t think we should become fixated on 95% confidence in every business A/B test. A result might still be useful to the company even if it doesn’t meet that threshold. The person running the test should choose the appropriate level.

Looking at each day as a separate test can make things clearer

At Buzzvil, we often graph results by day. Instead of treating two weeks as one experiment, imagine viewing them as fourteen daily experiments.

If B beats A every day, that consistency can make the difference convincing even before the aggregate calculation. The intuition is like flipping a coin: one heads has a 50% chance, but fourteen heads in a row is extremely unlikely. A single aggregate result of 3% versus 4% needs a significance assessment, while fourteen consistent observations of “A’s CTR < B’s CTR” provide a compelling pattern.

If the daily results alternate, the aggregate assessment described above becomes more useful. If B wins ten days and A wins four, B is probably higher overall even before deeper analysis. The genuinely difficult case is when B usually leads but A has an unusually large result on a particular day—an outlier.

In that situation, we usually:

1. Identify the users responsible for the outlier and assess whether including them in the sample is appropriate.
2. Investigate whether an external factor caused it, including the possibility of incorrect input data.

Breaking data down by hour, day, or week can reveal insights hidden in an aggregate. Sometimes the pattern is clear enough to guide the conclusion without further statistical analysis, which is why I strongly recommend inspecting it. One large test is useful, but looking at fourteen smaller periods can provide more insight.

That concludes this discussion of A/B testing. If you have questions or think something is incorrect, please contact elia.rho@buzzvil.com. I hope it helps with your own experiments. Thank you.