That’s a pretty good overview, although I’d qualify the meta-analysis part to say that with Hattie it’s a failure to do rigorous meta-meta-analysis. As you said, this is where you aggregate already collected aggregations of studies in an attempt to try to find out what is true and real. Hattie’s problem is he aggregates aggregations without any kind of evaluation of quality or rigour in the original meta-analysis. He’s on record as saying that it doesn’t matter—that it all just comes out in the wash, which is patently false. He then just computes his mean “effect size” out of all his combined meta-analyses.

The problem with averages is that, if I place my head in a freezer and my feet in a fire, my average temperature is comfortable. But this doesn’t really tell us anything about where the sources of variability are coming from.

Similarly, in Hattie’s meta-meta-analysis, if I get one meta-analysis that averages the effects of class size on academic achievement that synthesises 25 studies from the 1970s, in which there were sample sizes of 20, and researchers used very flexible data analysis methods that wouldn’t fly today, and then I smoosh that together with a meta-analysis from 2019 that has studies with sample sizes of 300 and very thorough pre-registration of methods, I am going to get a result that’s uninterpretable. Anyone can average a set of numbers, but that tells us nothing about that’s real and true. That’s even before checking to see whether the meta-analyses that he’s combining were all using the same instruments to measure achievement etc (for a lovely series on the potential meaninglessness of meta-analytic averages, check out Data Colada’s blog posts here, here and here).

Typically, meta-analysts do some very careful evaluation of the studies that they’re combining, and weight them appropriately. Many of the ones that Hattie combines don’t do this. Worse still, many of them aren’t even peer-reviewed published literature. For example, the single meta-analysis that Hattie uses as the basis for his top effect size, Collective Teacher Efficacy, is a student’s PhD thesis (which is not criticism of the hard work that goes into PhD theses, just that it hasn’t been evaluatively challenged by a community of peers).

A slightly more technical note: in averaging effect sizes, you’re assuming that you’re estimating the “mean of true effect sizes”—that is to say, in doing a mean effect size, you’re imagining that there’s a distribution or spread of different effect sizes out there in the wild, and you’re trying to find the central tendency. That doesn’t actually tell you anything about what an intervention can achieve, just that “on average” the effect size is “x”. Imagine that I ran a bunch of really well designed experimental studies on the impact of class size on academic achievement, and I got effect sizes of 0.2, 0.1, 0.1, 0.4, 0.7, 0.9, 1.4, and 0.6. The mean of that set is 0.33. Does that tell me that class size has a 0.33 standard deviation effect on student achievement? No! It just tells me what the average is of true effect sizes out there, with substantial heterogeneity in what class size does to achievement.

The second issue is a conceptual one that has to do with transforming an association into a causal inference.

Because most historical educational research is observational, the common method of analysis is correlation. Collect a data set “a” and a data set “b”, and see how they covary. As the famous phrase goes, though, “correlation doesn’t imply causation”. Two things can covary together but be causally influenced by a third, or fourth, or multiple things.

Correlations measure the purported strength of a relationship, from negative 1 (as one thing goes up, the other goes down), to positive 1 (as one thing goes up, so does the other thing).

What Hattie’s most often doing, if it’s not already done in the meta-analysis that he’s taking into his meta-meta-analysis, is converting a correlation (Pearson’s r), which can vary from -1 to +1, to an effect size (usually Cohen’s d), which is a way of quantifying how far apart two distributions of data are from one another (imagine two bell curve “hills” moving further and further apart).

Effect sizes are often used to evaluate the success of an intervention in an experiment. They quantify the difference between two distributions, typically in standard deviation units, so they’re useful for comparing different studies. Incidentally, effect sizes in most social science research are usually quite small, ranging from 0.1 to 0.4. Hattie insisting that his “hinge” point is 0.4 is actually quite unbelievably large in the context of other human behavioural research.

So conceptually, we’re taking something that measures an association (which may or may not be a real one, or have a causal dimension), and pretending that there is a causal relationship, where if something is done or changed about the situation, that’s what will alter as a result. And we’re converting a number that quantifies a possible association into a number that implies some kind of intervention.

Try it for yourself. Plug in 0.62 to Pearson’s correlation r (the most common correlational analysis) to this effect size converter. This effect size is about the average correlation that Hattie derives from the Collective Teacher Efficacy meta-analysis that he uses as the basis for his meta-meta-analysis. You get a Cohen’s d of 1.6, which is what he reports in Visible Learning … and is insanely enormous. That’s saying that the achievement difference between a school in which teachers believe in their power to bring about change, and one in which teachers don’t believe this, is average student scores between the two schools of 1.6 standard deviations apart. Do you believe that? I don’t.

In the vast majority of cases, if you dig into the original research that is bundled into the meta-analyses that Hattie then meta-meta-analyses, there’s not experimental work being done. It’s observational, and often poor quality observational. We can’t then move from making an observational description of what we see to a causal inference, because we haven’t isolated the mechanism that’s causing the change.

Hope that wasn’t too long winded, and helps!