Multivariate Testing

All you need to know about MVT​

Finding the perfect combination. That’s what multivariate testing is all about. A multivariate test is a test that simultaneously tests several combinations of several variables.

The idea is to modify several elements simultaneously on the same page and then define which one, among all of the possible combinations, has the most impact on the indicators being tracked.

Multivariate testing (MVT) helps test associations of variables, which is not the case with successive A/B (or A/B/C, etc.) tests. Unlike classic A/B testing, multivariate testing allows you to understand which combination of elements works the best for your visitors and their specific needs. Sounds appealing, doesn’t it? Learn all you need to know in this multivariate testing guide and try any combination of your ideas.

Table of contents

What is a multivariate test?

During an A/B test, you may not modify more than one element at a time (for example, the wording of a button) in order to be able to measure the impact. If you modify both the button’s wording and color (for example, a blue “Buy” button vs. a red “Make the most of it” button) and notice an improvement, how will you know if it was the change in wording or color that contributed to this performance? The impact of one change could be negligible or they each could have had an equal impact.

Multivariate testing looks to provide the solution. You can change a title and an image at the same time. With multivariate tests, you test a hypothesis for which several variables are modified and determine which combination from among all possible solutions performed the best. If you create 3 different versions of 2 specific variables, you then have nine combinations in total (number of variants of the first variable X number of variants of the second).

More articles on multivariate testing:

The history of multivariate testing

Testing methods like MVTs started back in the 1700’s. Scurvy was a major problem back then. Without knowing it, a British Royal Navy ship surgeon created the very first multivariate test in history, when he started giving sick crew members different solutions and treated them under different conditions: a high number of variables that, in the end, he could compare to see how these variables interacted with one another.

This multivariate testing led him to measure the effectiveness of each combination and find out the perfect treatment for scurvy: Citrus fruits, fresh air and lots of sleep.

What kind of websites are relevant for MVT?

Multivariate testing can benefit any website that has a purpose behind it. Because technically, the way of reaching a goal can always be improved. And so can any website. Some sites are aiming at lead generation, e-commerce sites are aiming at selling. Media sites, for example, could benefit from multivariate tests by improving editorial features, not a number of transactions.

Most websites do multivariate tests like:

  • Testing the different combinations of text and color of a call-to-action button.
  • Testing how text and visual elements on a webpage work together the most effective.

What types of multivariate tests are there?

There are 2 main methods for performing multivariate tests:

  • “Full Factorial”: This is the method generally referred to when we talk about multivariate testing. With this method, all of the possible combinations of variables are designed and tested over equal parts of traffic. If you test 2 variants of one element and 3 of another, each of the 6 combinations will therefore receive 16.66% of your traffic.
  • “Fractional Factorial”: as its name suggests, only a fraction of possible combinations is effectively tested on your traffic. The conversion rate of untested combinations is statistically deduced based on those actually tested. This method has the disadvantage of being less precise, but requires less traffic.

Why run multivariate tests?

There are three benefits to MVT:

  • Avoid performing successive A/B tests and save time since multi variant testing can be seen as performing several A/B tests on the same page at the same time.
  • Determine the impact of each variable in measured gains.
  • Measure the impact of interactions between different elements presumed to be independent (for example, page title and illustration visual).

Limits of MVT

The first limit concerns the number of visitors needed for your multivariate test’s results to be significant. By multiplying the number of variables and versions tested in your multivariate test, you will quickly reach a large number of combinations. The sample assigned to each combination will be reduced proportionally.

Where, for a traditional A/B test, you would assign 50% of your traffic to the original version in the tool and the rest to the variant, you will only assign 5, 10, or 15% of your traffic to each combination in a multivariate test. In practice, this often translates into longer tests and an inability to reach the statistical significance needed to make a decision. This is especially true if you test pages deep within your site with low traffic, which is often the case in order tunnels or landing pages for your traffic acquisition campaigns.

The second limit is linked to the way the multivariate test is defined. In some cases, it’s the result of an admission of weakness: the users don’t know exactly what to test and think that by testing several things at once in a multivariate test, they will eventually find a solution they can take advantage of. We then often find small changes at work in these multivariate tests. A/B testing, on the other hand, requires great rigour and helps better identify test hypotheses, which generally lead to more creative tests, backed up by data, with better results.

The third limit is related to complexity. Conducting an A/B test is often easier than a multivariate test, especially when analysing the results. You don’t have to do complex mental gymnastics to try to understand why a particular element interacts positively with another in one case but not in another. Keeping the process simple and quick to perform helps maintain confidence and rapidly reiterate on optimization ideas.

If you think A/B testing is a better fit for your use case, don’t forget to download our A/B Testing Ebook.

Multivariate test ideas and hypotheses

The key to a successful MVT approach is a strong hypotheses for every element tested. This hypothesis should later be implemented in the different test modules and combinations of your MVT.

In order to create a strong MVT hypotheses you must:

  • Clearly identify the question you are interested in answering with your MVT
  • But note: A hypothesis is a statement, not a question. It is a very clear testable prediction about what will happen, if certain changes are being made to a website
  • Make it clear and link your prediction to a problem that has identifiable causes
  • Mention a possible solution

More articles about creating MVT test hypotheses:

Testing sample size

In order to test a MVT hypothesis, you need a larger sample size. Think of your multivariate test as several parallel A/B tests and increase the number of tested visitors accordingly.

In short: A good multivariate test requires enough website traffic to test multiple variations simultaneously. Therefore, the required sample size should never exceed your level of website traffic, unless you want to wait forever for your test results to be valid.

High traffic is important for multivariate testing
Multivariate testing requires more traffic than A/B testing

What is the ideal length for my MVT?

There is no universal answer to this question, but to give you an idea, here’s a simple calculation. Let’s say your website has 30,000 visitors a day and about 5% of the visitors convert and you want to test three variations, your test should run for 11 days. If your website has only 5,000 visitors a day and an average conversion rate of 2%, the required number of tested visitors per variation is 78.039, which will require your test to run 468 days.

If your average number of visitors is very low, multi variant testing may be inappropriate for you. Check out A/B testing instead! Additionally, here are six techniques for getting started with testing with low traffic.

Want to know where we took those numbers from? Check out our free sample size calculator!

Our online sample size calculator helps you calculate the minimum sample size as well as the duration of your tests based on your audience, your conversions and other information such as the Minimum Detectable Effect. This helps you increase your confidence level before making any decisions to improve your conversion rate.

Tips and best practices

Here are some tips that will help you set up your first multivariate tests and avoid common mistakes.

1. Choose a strong testing tool

Multivariate testing is often assumed to be very technical, so we suggest you go for a testing tool which keeps it simple for you to use. AB Tasty, for example, makes it easy for any marketer to jump into multivariate testing and helps you gain valuable customer insights for you to make the right decisions.

2. Form a good testing team

In a strong CRO team, different tasks should be clearly defined and distributed. For example: A “conversion manager“ could be lead the team and be in charge of QA. Another team member could be responsible for a first analysis of your visitors‘ behaviour and the status quo. A designer could take care of aesthetic modifications on your website and a technical profile (JS and CSS developer) should be responsible for the implementation of advanced tests. A data scientist could be in charge of evaluating your results in the end. One person can be in charge of different tasks at the same time if they have all skills necessary and sufficient time to meet all tasks.

3. Have a plan and clarify a timeline

It’s all about a good structure. Before you start creating your multivariate tests, clearly define what elements should be tested, why they are being tested and define a time frame. Knowing when the MVT results are needed will help you work more efficiently.

4. Set targets and define success and failure

Set annual targets that can be adjusted each year, for example the number of campaigns launched. Quantitative measurement is easy and precise. For your multivariate testing campaigns, clearly define what makes a test successful. Note that even a failed test is worth something because it helps you understand what doesn’t work and needs to be changed in the future.

5. Create a knowledge database

Keep track of the most important things you learned, save testing knowledge in a database and avoid recurring failure in the future. Once knowledge is acquired, it should be made available to everyone in your team. It will also make the onboarding process of new team members more efficient

6. Include all third parties necessary

Workplace loneliness is a real problem. Your CRO team should not be working in an isolated way. Let others know what your CRO department is working on and spread the word about testing results. Also, be open to new ideas and input from others outside your CRO team!

7. Identify and test different audience segments

In your multivariate testing campaigns you may determine returning visitors prefer a different website design than new visitors. Innovative tools like AB Tasty will recognize and automatically suggest visitor segmentation.

Articles on best practices worth a read:

Examples

Looking for ideas for your very own multivariate tests? Below you’ll find some links to a few examples and testing inspiration:

Multivariate testing softwares

Make sure you use a tool that actually addresses the problems you need to solve. When it comes to improving your website’s conversion rates, in a wide-range optimization process there should be much more involved than testing alone. Therefore, choose a tool that helps you fully understand user behavior. We recommend you use AB Tasty, as it offers you numerous sources of information you can use to gain this fuller picture:

Other forms of testing

There’s much more than multivariate tests out there! Here’s a list of other testing scenarios:

  • A/B/n Testing: Build and compare two or more variations of the same element
  • Split testing: Redirect traffic to one or several URLs. A perfect fit for new pages hosted on your servers
  • Multi-Page Testing: Display changes consistently across multiple pages (Funnel Testing)

Refer to this article to learn how to choose between this different testing methods.

Website optimization is not limited to testing. You can use advanced audience segmentation personalization to deliver tailored experiences across every customer touchpoint and much more.