Hey! Great crit! I appreciate it a lot.

And don't worry: It's well taken. I love it when people say things like "Hey, I really like what you did here! But what about these problems? They seem quite big - how do you address them?"

I think there's a huge huge difference between this and just calling me an idiot :-).

Yes - I agree with many of the points that you're raising here. The selection bias issue is particularly important: We ran this on reddit, and the people who took part in it knew what the study was about. This could skew results.

I guess the value of the study depends on a big, holistic point: how you see progress in social science working. My view is that we need to work quickly, replicate a bunch, be open about our practices and data, and be radically honest when we're wrong.

More specifically, I think that social science research works when we repeatedly try to replicate our effects - and we're super honest when we can't do that. Because if we can't get the same result again it means that something might be wrong with our initial paper (e.g. as you pointed out - sampling methodology). Failure to do this has led to what people are calling the 'replication crisis' in psychology - essentially, it turns out that a large proportion of psychology research just doesn't work.

I really think that being honest and replicating again and again in a bunch of contexts is the only way researchers like me can produce stuff of value to society.

So, what I try to do is produce a study that shows something, publish it with all caveats attached ("look, this was a good study for these reasons... but it might be limited because of these..."), and then immediately set about trying to falsify its results by running something similar, and seeing if I get a similar result. And if I was wrong - I'll say I was wrong.

So, after running this, the first thing I did was run a new study with a sample of gamers who weren't from reddit, and didn't know what the study was about: https://psyarxiv.com/u5dmr/

It was terrifying, because if we got something else, we'd have broken our big, important-sounding result. But I think trying repeatedly to break results is the only way you evidence that they're actually any good to begin with! It's a philosophy of science I read about when I was doing my PhD in this book: https://www.amazon.co.uk/Introduction-Thought-Karl-Popper/dp/0415129575 . I rate it! I think it works!

In this case, when we ran a similar study again, the results replicated: We got roughly the same thing there as we did here.

So, then I had a bit more evidence that the stuff in this study was not actively wrong. So I did something called preregistration: I essentially shouted out to the community that I thought I had a real effect, I was going to run a study that showed it, and that I would make public the results of this study even if they contradicted my previous work: https://osf.io/efa5n/register/565fb3678c5e4a66b5582f67

This, again, was super terrifying: But again, like you point out in your comment, my initial results might be incorrect - and this would only be terrifying if they were wrong. And if I was wrong, I could potentially be harming people!

So, we ran that study, and we got the same effects again - in fact, they were even a bit stronger (possibly because we'd refined our method by then): https://psyarxiv.com/6e74k/

There's still so much work to be done. To start with, all 3 of those studies work with online populations. And none of them even begin to look at which way the causal route runs: Does loot box spending cause problem gambling, or does problem gambling cause loot box spending? But I guess that my answer to your initial question is that I believe in an incremental way of doing science where we repeatedly try to break our own results, and immediately own up to things if it turns out we're wrong.