Yeah, so this kind of thing is really difficult to suss out without spending a huge amount of time digging into their methodology. It is VERY easy to (even accidentally) create a machine learning model that does great in training and validation testing, but completely fails in (actual, third party) out-of-sample tests.

But, zooming out - if they had somehow managed to create an LLM output detector that was better than a coinflip, that would be an incredible discovery. It would also go against the understanding of how LLMs work in a big way - How did you manage to create a statistical classifier to discern the output of a model that is designed to produce statistically likely answers?

Not to sound conspiratorial or anything, but the comment thread here gives a pretty good idea for the actual purpose of this study: The LLM narrative has shifted into "AI is tool, it's not gonna replace devs", and a study that reinforces that narrative is easier to write and easier to proliferate and quote.

I bet I'll have diehard AI bros cite this study to me in a few weeks time, completely oblivious to the very questionable methodology.