What is Multivariate Testing and Learn How to Test Your Pages?

Multivariate testing is a technique that benefits approaches of engineering and statistics to choose between different options, or variates as commonly known. Remember the days when you made your toys fight with each other to select the strongest one. Now think that things got elaborated and your toys have removable parts and you try different combinations that have different arms, tails, or feet. This is basically multivariate testing for the winner toy. Marketers apply multivariate experiment to optimize their sales.

Multivariate test vs A/B testing

Over the years A/B testing have become the standard in the market. It is mainly because it is easy to analyze. You have two alternatives. Put them at display and let the visitors decide.

Remember the toy example? A/B testing in that situation would be selecting two complete toys that you think is the strongest. So one of them would be the winner. Back to the digital world…

You are trying to decide between two different headlines for your charity funding page. One saying “Donate now to help save children’s lives” and the other one saying “Show your support for world’s children”.

A/B testing would show 50% of the visitors the alternative 1. The other half would see the second alternative. Observing the conversion rates and donation amounts, the one with better results is chosen.

So how does multivariate testing (MVT) change the game?

Assume besides having 2 slogans, you also couldn’t decide where to place it. Or even which picture to put on the background. This is where multivariate testing becomes the solution. Instead of having 2 alternatives, we need 8 (a multiplier of 2 for each case). Multivariate testing creates 8 different pages to show to visitors. Each having 12.5% of the general traffic. By this way 3 different variables can be tested with only one test and come up with optimized combination.

What are variable and variates?

Before we continue knowing the difference between variate and variable is crucial. Variables are the properties we are testing. The headline, font, background color, and button display are each separate examples of variables. Variates are the alternatives we show to our customers.These are formed by different combinations of selections in variables.

What are the types of multivariate testing?

There are three types of MVT.

Full factorial testing: This is basically testing every variant you have. This is the most popular technique as it has no assumptions but all-covering numerical data. Biggest drawback is required traffic. As the number of variates increase, more traffic is needed to make meaningful results about different variates. Think of 20 variates each having 5% of the traffic.

Fractional factorial testing: As the name suggest this technique include some of the possible combinations. Most known technique belongs to famous Japanese engineer and statistician Taguchi. His methods enables us to test high number of variables with fairly low numbers of variates yet having enough information about every variable.

This method almost eliminates traffic requirement problem but leans on some some statistical assumptions and errors. The very same methods will show you the variation of parameters used in MVT.
Adaptive multivariate testing: The newest method of MVT uses real-time data to come up with the winner combination.

Why should I use MVT?

• Best thing about multivariate testing is the precision it has. You will end up with clear views on how to arrange your design. This precision is reached by element-by-element testing of MVT.
• Except some types of MVT, it contains no assumptions. MVT technique deals with each of the changes, which enables it have full numerical data.
• Interrelation of different features or objects is hard to catch.Not with MVT. Multivariate testing presents information on how thing go well together.
• Speak with data not with gut feelings as a marketing professional.

What are the drawbacks of MVT?

• The biggest problem with MVT is its high traffic requirement. If you have 10 different combinations, each having 10% of the traffic, you need large volumes of traffic to have meaningful and trustable outcomes. So the advise is not to use it once you are new in town and don’t have enough traffic.
• As this technique is mainly used for correlation between different aspects of a page and aims small but high impact changes, MVT is not an option to begin a webpage. A/B testing will be your welcome gift. Same thing goes for if you think of a whole redesign.

• One last drawback is MVT takes time! Fairly long time. In emergency, don’t use MVT.

Dos and Dont’s of Multivariate testing

There are certain golden rules for doing MVT you must not miss.

What not to do?

Don’t change excessive number of variables. Advised gap is between 3 and 5. As you increase the number of changed variables, shared traffic volume diminish by half.

What should you do?

• As you many combinations, some are guaranteed to be ridiculous.A combination having a banner showing “Free admission” and a headline saying “50%discount” as a gift will contradict. This will confuse the customer and will reduce your credibility. So check every combination for fallacies.
• Estimating your traffic and timespan you can spare for the test before starting your multivariate test will clear your vision.These will even help you to decide whether to apply multivariate testing. Then, some more like how many variable to change or how many combinations to put on the test.

Now you can try MVT for the sake of deciding better webpages.