A/B Testing, Split testing and Hold-out testing

A/B or AB testing refers to testing two different versions of a page or a page element such as a heading, image or button.

The alternatives are served alternately with the visitors to the page randomly split between the two pages. Hence, it is sometimes called live split testing. Changes in visitor behaviour can then be compared using different metrics such as click-through rate on page elements like buttons or images, or macro-conversion rates, such as conversion to sale or sign-up.

AB testing is aimed at increasing page or site effectiveness against key performance indicators including click-through rate, conversion rates and revenue per visit. Since it does not consider combinations of variables tested, for best uplift multivariate testing is increasingly used.

Although the term implies only two versions of the creative, typically the original (champion) vs Challenger, it’s often more useful to compare the original control to two or more alternatives. With two alternatives to the original, it is known as ABC testing.

Split testing is similar since different groups within an audience receive alternative communications with different creative, offer or frequency to determine which is most effective to inform the communications used for future campaigns. 

Direct marketers sending costly large volume mailings would often test different creative and offers using split-testing to test different elements of the communication before sending out the best performing offer to a larger audience. They would also use hold-out testing where a control group are excluded from a campaign to understand the incremental affect of changing communications of those who receive communications against those that don’t.

These techniques can also be applied for email marketing or other forms of customer communications.


Welcome to my personal site where I blog about the latest developments in digital business to help students, professionals or lecturers using my books in their studies or careers.

Visit my main Smart Insights Digital Marketing Strategy site to download planning templates, strategy guides and daily updates on the latest developments.

I also write about assistive technology to help others teaching the visually impaired as part of my volunteering work for charity AbilityNet.

My latest books

Full details on Amazon:

Digital Marketing: Strategy, Implementation and Practice
8th edition

Digital Marketing Excellence
6th edition

About Smart Insights

Dr Dave Chaffey is co-founder and Content Director of digital marketing advice site Smart Insights. We’re a publisher and learning platform that helps our 150,000 active members in over 100 countries plan, manage and optimize their digital marketing activities by applying the actionable advice in our planning templates, guides and interactive e-learning tools.

Free members can download our free templates and benchmark their digital capabilities.