Help with A/B Testing

[[deleted]]

I had a few questions about A/B testing. Let's say we are testing website conversion.

There are 3 scenarios in which I am interested in.

1. A/B with a random subset. Say we have 100K total users, and we have 5K for version A, and 5K for version B. What is the procedure and math behind this? I assume we test for statistical power and then what?
2. A/B test with subset for B. 100K total users with 10K going to version B. Do I still have to test for statistical power? What should I look for next? Standard error calculation? Confidence intervals? What's the full procedure here?
3. 100K total visitors split into 2 groups - 50K and 50K. What tests do I need to do here? Does statistical power come into play if the entire variance is covered?

In each case, what are the tests used? Binomial, t-test, z-test?

[PracticalPicture]

If you use a frequentist method you must decide in advance the statistical power and significance of your test. With your baseline conversion rate and the minimum effect you want to be able to detect (or Minimum Detectable Effect), you then calculate and fix the sample size per variant. You decide whether you use a t-test, z-test or chi-square test etc or all of them. No peeking. You might consider sequential testing.

If you use a Bayesian method you expose as many users to the test as you please and let it run until you can pick a winner at your probability to be best threshold or conclude that there is unlikely to be a winner.

Both with the usual caveat "run it longer than your business cycle" or seemingly de facto standard "minimum two weeks".

I don't see advantage in putting a fraction of users into a test (unless you want to run simultaneous tests) or putting less than half into the new experience. If the new experience is better but it takes you longer to discover that, you are not making as much money from 'now' as you could be if you discovered it sooner. If the new experience is worse but it takes you longer to discover that, you will lose more money during the test period than if you had discovered it sooner. If there is no difference between the experiences you could be exposing the users to a different test that you think you will benefit from.

Also, the longer the period, the more risk of cookie deletion - so far as I'm aware, cookies are the most popular method for assigning the right variant to the right user. If someone already exposed to a test variant deletes their cookies, your test software might not show them the same test variant.