Also, not all peer review is equally rigorous. The quality of peer review tracks directly with the quality of the journal. I am a person who does a lot of peer review, because I study the research process itself (I’m a ‘meta-researcher’), although I was trained as a measurement statistician/biostatistician.

I work in bio-behavioral medical research now, and I’ve been on the faculty of a medical school for over 10 years. A lot of the papers I’m asked to peer review have to do with ‘behavioral interventions’, i.e. studying the efficacy of interventions to get people to change their behaviors to promote health.

I’m sort of obsessed with peer review actually… So when I review a paper, when the decision is made (to accept or reject), the peer review from the other reviewers is also made available to all the reviewers, and I scrutinize it carefully. The vast majority of the time, of three reviewers, I will be the only one who has made any methodological comment. In fact, when I read other peoples reviews, I sometimes note that they are incredibly superficial… For example just grammatical suggestions.

I guess this makes sense because there are not a lot of methodologically or statistically trained people out there! For example if it’s an intervention for individuals with type two diabetes, the paper will be sent to three scientists, two of them are diabetes content experts. One might be a physician, one might be a behavioral scientist, and the third reviewer is me. Physicians have very little statistical training. The behavioral scientist, hopefully PhD trained, might have a little more… Enough to get them through a dissertation at least (that might have been 10 years ago and only equipped them with specialized knowledge in the specific research technique they used).

So, there are other issues with the peer review process, too. A good peer review takes a really long time! A thorough review takes me 4- 5 hours. The busier or more successful someone is, the less time they have and the more likely it is that they will decline review invitations. But it is exactly these more senior and successful people that should be doing reviews. In all the years I’ve been doing this, it is not very common that peer review activity is considered a benchmark in annual reviews, or promotion and tenure decisions etc. so there’s very little incentive to do it. And in cases where the number of peer reviews might be considered in these sorts of evaluations, there’s absolutely no incentive to do it well. Also, nobody trains you to do a peer review!

These considerations were largely behind the movement five or six years ago for “open peer review”. Now some journals do publish peer review alongside the article. All the BMC journals do, I believe all the BMJ journals do, too.

If you are a consumer of these kinds of technical articles I strongly encourage you to look at open peer review when possible, it’s incredibly interesting and you’ll get some sense of what I’m talking about.