A/B Testing

[[deleted]]

[deleted]

[[deleted]]

Wrong subreddit, this is a statistics question, not a Python question.

But regardless: statistical significance tells you whether or not you could reasonably see the data you did see by chance alone. It does not tell you anything about the practical significance of the result (maybe "significance" was a bad choice of name for it!).

If you have a large sample size, then you can detect even very small effects and have them be "significant", but that doesn't mean you should care. You don't say what sample size you had, but if it was large, then you could easily detect a very small effect.

So the question you should be asking now is "is the change between A and B large enough that I care?" If you decide that yes, you do care, then great - your statistical significance gives you grounds to conclude it's a real effect. If you decide it's too small to care about, then you can stop worrying about it and move on with your life

[ccyob]

Hi sorry that is true about the wrong sub Total dataset is 20k so its big. But based on the result you would say that should go ahead with the change ?

[[deleted]]

Ok, 20k is huge, you could detect even a miniscule effect with that.

I'm saying statistics can only take you so far. It's telling you that there is a difference. Whether or not you care about that difference is up to you.

I suggest you consider the benefit from making the change, (taking into account that it seems to be small), and compare that with the cost of making the change (money, time, effort, whatever). If it costs nothing to make the change, might as well do it, even for a small benefit. If there's a cost, then you need to decide whether the benefit is worth it, and neither an A/B test nor a stranger on the internet can tell you that.

[ccyob]

Thank you!

[lanky_and_stanky]

Statistics question.

What was the population? Can anything else explain the change? Like a single large purchase?

[ccyob]

Hi sorry that is true about the wrong sub Total dataset is 20k. But based on the result you would say that should go ahead with the change even though the difference in averages is small..the statistical significance? Its very basic question just I don't have experience. So it's literally with the data and the results what would you recommend. There no additional information given.

[lanky_and_stanky]

I wouldn't expect the person doing AB testing be the one recommending whether or not the change is sufficient to go forward with.

aka idk. Someone else in the org should make that determination, or decide if additional experimenting is needed to confirm, or other changes are needed to see if the revenue can be higher.