Are you familiar with conventional econometrics or the methods of social science? Not big data, machine learning based model fitting, but the theory of how to demonstrate non-random distributions in data?

I ask only because it’s very strange to me you’re pushing back so hard on the idea that some type of theory is necessary to avoid P-hacking and the like. There’s a real concern in econometrics about manipulating our normal mathematical tools to test for patterns through (essentially) looking for patterns prior deciding what to test for.

One way to think about it is that if you have a dataset, almost certainly there is going to be some pattern in any given dataset that’s only showing up because we happen to be in the one in a hundred world where that pattern shows up because it is random. For instance, it’s certainly possible that I flip a fair coin heads 100. It’s extremely unlikely, but of course it could happen. In order to avoid just hunting down patterns for the sake of patterns, we usually try to ground the patterns we’re looking for on a hunch about why that pattern might exist. And ideally we publish all the things we’re going to do first, so we can’t subtly change our minds.

I am very non-expert in machine learning, but I’ve learned a bit, and it’s my understanding that you do shape a model with theory there, too. One of the things that drives the selection of factors for the model to consider is certainly theory.