AB and Multivariate Testing for Success: Part I

Even when research and best practice inform your changes, there are so many variables at play, that you can’t always predict the outcome.

So how can you make smart changes to your website that will achieve your business goals? This is where A/B and multivariate testing come in to help you make confident and informed decisions about the development of your website using real world data. By testing changes with your audience, you can make evidence-based, strategic decisions.

Website testing is a broad subject with testing encompassing everything from technical validation to quality assurance, but for the purposes of this article we’ll focus on website testing from a conversion rate and performance enhancement perspective.

In this context, website testing is the process of exposing members of your audience to proposed website changes and analysing the results to determine whether the changes are worthwhile and valuable.

There are three key benefits to testing website changes instead of just making changes based on 'gut-feel' or even best practice.

• The data-driven path to more conversions
  A/B testing helps eliminate guesswork from website optimisation. Instead of relying on opinions or assumptions, you can make decisions based on concrete data and statistics.
• Reduced Risk
  A/B testing lets you minimise risks associated with major website changes. Instead of making sweeping alterations blindly, you can incrementally optimise various elements for better results.
• Continuous Optimisation
  A/B testing promotes an iterative approach to website improvement. You can keep refining your website over time, leading to gradual but substantial enhancements.

One of the most important aspects of testing is the fact that it runs both the test variations and the control (the original design) at the same time. This reduces the potential for external factors to skew test results.

It can be tempting to skip testing entirely and push changes live with the intent to assess performance data by direct comparison to site analytics before the change. The disadvantage of this approach is that it isn't comparing apples to apples. Factors impacting performance may have been in effect before the change went live that ceased to exist after, or vice versa. This impacts your data and makes comparisons inherently unreliable.

To use an extreme example, imagine setting your new design live the day before news of a recession hits and people instinctively tighten their purse strings. The new variation may still be better than the original but seemingly show worse performance because it was live during a period of reduced consumer spending, while the original wasn't.

Showing members of your audience both the original design and any test variations over the same time period removes that uncertainty.

So far we've spoken about website testing in general terms. However when the subject is typically discussed, it’s in the context of A/B and multivariate testing.

Both forms of testing have the same aim–discover the best performing variation of a page or application design–but they operate differently and suit different contexts.

A/B Testing

A/B testing (also known as split testing) is the simplest form of testing. In A/B testing you test 'whole of page' variations. The variations can be minor (such as a change to a headline) or major (an entirely new page layout with brand new content). The key thing is that each design is treated as its own entity for the purposes of testing.

Is ideal for websites with low traffic or where there’s a fast turnaround required. That’s because A/B testing requires less traffic to reach statistical certainty (the point at which you can be confident the winner will remain so when set live).

Is ideal for testing wide-scale changes to page and screen designs.

Is very useful for testing isolated changes.

Is the perfect starting point for most new test regimes as it allows you to quickly establish a high performing base before moving on to more incremental changes.

Multivariate Testing

Multivariate testing differs from A/B testing in that it is more granular. Rather than test multiple pages, multivariate testing focuses on multiple variations of specific page elements and exposing users to different combinations of these elements. This kind of testing goes beyond telling you which version of a page produces the best results, and tells you which elements on the page have the biggest impact, helping you optimise at a far more granular level.

Is perfect for making incremental changes to already high-performing pages

Requires a lot of traffic since mixing and matching different element designs means the real number of variations can be quite high. For instance, testing three elements of a page, each with three variations (including the original design) means 27 variations to be tested

Is tailor-made to testing multiple page changes at scale

Helps you understand the impact of different elements on page performance, both in isolation, and together.

The first thing you need to do when getting ready to run tests on your website is a testing platform.

In the past an easy service to point to was Google Optimize, but the search giant has announced its intention to close this service on September 30, 2023, leaving a market gap for affordable, newbie-friendly testing solutions.

The following are some of the more popular platforms out there. Be sure to investigate each with reference to your unique situation. Does the platform support the tests you want to run? Do the available plans support the level of your website's traffic and the level of testing activity you want to engage in?

The following are three of the more popular tools in this space and are presented without recommendation. If you need help in choosing the right platform for your needs, reach out to us for assistance.

• Optimizely
  Optimizely is one of the longest running testing platforms available and is considered the leading platform by many conversion rate optimisation specialists. Strong data visualisation, excellent third party tool support and retroactive filtering are just some of the features that keep Optimizely front of mind for many who make testing their living.
• Visual Website Optimizer (VWO)
  VWO is the other big player in the website testing space, having come out around roughly the same time as Optimizely. VWO's popularity is thanks to simple test set up via a WYSIWYG text editor, and the availability of other CRO-focused tools such as heat mapping and online surveys.
• AB Tasty
  Despite not entering the market until after the big players already mentioned, AB Tasty has developed a solid reputation as a testing and personalisation tool. It's considered an excellent starting point for many businesses thanks to reasonable pricing and a platform that facilitates a wide variety of test configurations.

The above three platforms are by no means the only testing platforms out there. Depending on your situation, you may find one of the above services perfect for your business, or your ideal platform may be one of the many other service providers we haven't mentioned. The most important thing is that the platform you choose:

Has the capabilities to run the tests you want to run

Is capable of being integrated into your website platform

Is easy to use

Can support the levels of traffic you need at a reasonable price.

A/B testing is an invaluable tool for enhancing website performance and user experience through evidence-based decision making. By pitting multiple versions against each other and measuring their impact on user behaviour, businesses can optimise their digital offerings for better engagement, conversions, and overall success.

In Part II of this series we’ll give you a step-by-step guide to running your A/B and multivariate testing, so make sure you’re following us on LinkedIn to get your hands on Part II as soon as it’s released.