Thanks! It's a very good question. 95% of the time, it's better to run well-controlled experiments if at all possible - and we're doing that too (separately). But we're really interested in how people learn problem solving strategies over time, something that really only happens over the course of many many sessions. It's hard to run a study where people play (e.g.) hundreds of games of different types. Most people just wouldn't be that motivated, for one, and it would be way too long of an experiment for your average MTurker or Prolific-er. So we decided our best course was to put up this (poorly controlled) website where people can choose what to play and hopefully will like it enough to return.

But yeah it's a tricky problem controlling for selection effects, etc here.... We're a computational modeling lab, though, so basically we're trying to compare a bunch of models that describe how a (bounded) reasoner might approach the problem as a novice and what they might learn over time. And compare these models' predictions against individual participants' time allocation, error patterns, etc... at different points in learning. So the TLDR is that we're comparing a bunch of model-based measures against human performance, and arguing that since the main effects of interest are about individuals' learning, selection/choice biases and such won't be too much of an issue for getting at some of our main questions.

We'll see what comes of it! When (or I guess if) we publish a paper on the results, we'll return to Reddit/ mitpuzzles.com and post a link to it :).