A/B testing primer

A/B testing is a controlled way to stop guessing: show version A to one group, version B to another, then measure which one gets more clicks, leads, or sales. (shopify.com) In marketing, A is usually the current ad or page and B is the changed version, and many teams split traffic evenly so each version gets a fair comparison. Google Ads says it recommends a 50% experiment split for the best comparison in custom experiments. (shopify.com) (support.google.com) The core rule is to change one variable at a time, such as a headline, image, audience, or bid strategy, so the result can be tied to one decision. Meta says tests with more than one variable can compare two ad sets, but they do not isolate which change caused the lift. (shopify.com) (facebook.com) Teams usually pick one primary metric before launch, such as click-through rate, conversion rate, cost per conversion, or revenue per user. Statsig’s documentation says experiments need a stated hypothesis, a randomization unit, and primary metrics before results can be read correctly. (docs.statsig.com 1) (docs.statsig.com 2) That process has become standard across ad platforms because both Google and Meta now build testing tools directly into their ad systems. Meta says its A/B test feature can use an existing campaign, ad set, or ad as the template for a test, while Google shows statistical significance and estimated performance differences inside its experiments dashboard. (facebook.com) (support.google.com) The hard part is not launching a test but waiting long enough for enough data. Google says small traffic splits can leave an experiment inconclusive, and Shopify says many A/B tests run at least 14 days, with longer windows for lower-traffic sites. (support.google.com) (shopify.com) The statistical check is meant to answer whether a result is likely real or just noise from a small sample. Google labels this as statistical significance in Ads experiments, and Shopify describes it as a sample-size question that many testing tools calculate automatically. (support.google.com) (shopify.com) A clean test also depends on random assignment, which means users are sorted into groups by rule instead of by marketer choice. Statsig says the randomization unit, such as a user or device, is a critical field because analysis works best when the reporting unit matches the assignment unit. (docs.statsig.com 1) (docs.statsig.com 2) For performance teams, the payoff is practical: test a new headline against the old one, keep the winner, then move to the next variable. That is the basic discipline behind the A/B testing primers now circulating among marketers who want to trade opinion for measured results. (shopify.com) (docs.statsig.com)