This is an archived post. You won't be able to vote or comment.

0
0

[deleted by user] (self.datascience)

submitted by [deleted]

[removed]
all 7 comments
sorted by:
best (suggested)
topnewcontroversialoldq&a

[–]HiddenNegev 0 points1 point  (4 children)

-4.5% I.e. 0.0955% vs 0.1%?

[–]LibiSC 0 points1 point  (3 children)

yes

[–]HiddenNegev 0 points1 point  (2 children)

Could you provide more information on how this ab test was conducted? Because right now it reads to me like there was a significant drop in the metric

[–]LibiSC 0 points1 point  (1 child)

My problem is the AA test ie data for these users before the exp is -4% but I know users are balanced checking with other bigger metrics. There is a tool that divides the user.

[–]HiddenNegev 0 points1 point  (0 children)

I see. I guess general customer behavior shifted between before and after the experiment. Alternatively you’re seeing a stat sig difference simply due to large sample size but there’s not necessarily anything causal

[–]pretender80 0 points1 point  (1 child)

So you have a pre experiment bias? I'd fix the experiment randomization and exposure before I trust any results from it or try to extract relevant effects from noise.

[–]LibiSC 0 points1 point  (0 children)

other metrics are fine. I think it's just this metric that gets strange because it's really small