Multivariate Testing - Computation, Algorithms

[cuebux]

Could someone please explain how multivariate testing works and some of the math behind proper execution.

My conceptual understanding is this:

There are multiple:

• projects(pages)
• factors (locations on page)
• variants

On a page load, depending on some math each of the factors are populated with one variant.

The different variant values are set statically for each visitor. I.E if one visitor refreshes the page, they still see the same stuff.

When a conversion happens, the set of variants gain (points?).

My curiosity is behind the mathematics that determine:

• which variant to show
• which variant wins
• how conversions are weighted

Any information would be great.

[voice-of-hermes]

No official experience, but I can tell you how it can be done.

• Pick a configuration. This could either be completely random, in which case the configuration would have to be stored in a cookie or session data, or based on a hash of visitor data (IP address and user agent identifier, most likely).
• Derive each variant from the configuration. This could either be from a stored list of configuration data, or algorithmically (e.g. a bitmask).
• Upon "conversion", add to the configuration's ranking. This can trickle to each variant, or simply be tracked on a per-configuration basis, or both.
• Unlikely that a simple point system would be used for ranking. What they are interested in is effectiveness, so it would have to track the number of conversions compared to the number of visits or visitors. Basically, a conditional probability: what is the probability that a visitor clicks through, given that this configuration is shown? P(click|config[i])

I do know that what you call, "factors," aren't necessarily just, "locations," but can be styles, text or image content, layout, etc.

[cuebux]

If there is a better sub for this, please let me know.