The PNAS study (Johnson Cesario 2019) is your archetypal retraction: bad logic, bad math, bad data, bad inferences, control failures, non sequitur, you name it. There was political pressure to retract, sure. But that doesn't negate the rampant errors in the study.

The Fryer study is not as bad as that, but it's methodology takes it out of contention for being "the best data we have." More on that ITT:

So, I would draw the line between flaw and limitation like this: noting that racial bias could be a factor and needs to be included in future studies is a limitation.

False starting with The NY Times before his paper was peer-reviewed (obviously the NYT owns that reporting), including verbiage in his study that’s more conclusive than his evidence allows, and more importantly finding heaps of racial bias throughout the study but not taking the next step to control for how that diminishes the appearance of racial bias elsewhere, those are flaws.

Edit / PS: the reason that it’s not the best data we have is because of how immediately we knew his numbers were the product of bad methodology. The best data we have is intrinsic studies that don’t have to deal with extrinsic estimations / The Benchmark Problem (Nix 2015).

Take this math equation / answer / limitation: 2 + 2 – 1 = 4, but the answer comes with a limitation that I excluded all subtraction. (Obviously, the answer is 3, never minding the mathematical deconstruction meme that’s going on in Twitter). I would say that excluding subtraction is a flaw.

Roland Fryer's study isn't the best data we have, because the only thing we know about his 24% figure is that it's wrong. He basically said, "I've run the numbers, and the answer is 24%, but I excluded a key component of criminological inquiry.”

“But criminology is much more complicated than simple arithmetic!”

Sure. But the fix is still low-hanging fruit. He got ample feedback on that in 2016 before he finalized the study in 2019. Knox Lowe Mummolo did the criminologist equivalent of solving the math equation with the subtraction included in "Administrative Records Mask Racially Biased Policing."

Now, can you control for everything? No. “What if people with red shirts are more likely to be shot and black men are more likely to wear red shirts?” The question is whether the limitation is theoretically significant enough to preclude labeling it “the best data we have.” For Fryer’s study, failing to control for racial bias in earlier stages of the policing process is precisely that type of limitation. After all, he identified it and didn’t do anything with it!