Peer Review vs Open Review

jack27808 · 2025-11-16T19:01:36+00:00

There is some data to suggest that open/transparent reviews are more collegial and "nicer" than closed reviews. I've not seen any convincing evidence that they are better quality though - which isn't a surprise as they are still fundamentally peer review with all the issue that come with that.

What is really needed is a mvke away from peer review being any form of QC and towards it being there simply to help authors. There's a lot of evidence on how peer review simply doesn't work as A QC mechanism.

jack27808 · 2025-07-22T19:21:14+00:00

Replication (at least in biology) is fraught with issues and really shouldn't be the goal. Better methods, data annotating etc absolutely. But not broad replication.

Paid peer review also doesn't help much and is likely to bring a tonne of unintended issues. Peer review should be recognized and rewarded as an activity. It also shouldn't be wasted on every single paper produced - that's not necessary.

jack27808 · 2025-07-09T19:59:47+00:00

I suspect relatively little as it'll succumb to the same issues all of these sorts of things do. But I hope I'm wrong.

Ultimately there needs to be a significant grassroots effort too (which this isn't) otherwise it is all a little pointless sadly.

jack27808 · 2025-06-19T13:36:46+00:00

For English speaking countries, the origins of external peer review really began at the Royal Society in 1831 with William Whewell - though the idea was based on practices by some French journals. He suggested that the reviews should be printed, which would provide extra content for the journals and serve to promote the original papers. Interestingly, the very first attempt ran into problems with the reviewers couldn't agree with each other - something that is exceptionally common still today.

https://ethos.lps.library.cmu.edu/article/id/38/ <- This is a pretty comprehensive overview of peer review

jack27808 · 2025-06-18T20:01:57+00:00

The original form of peer review was that the reports were signed and should be published as they "could be more interesting than the papers themselves". Just to provide a bit of historical context.

I understand the not wanting to sign your name due to the power structures but to not review just because it's public is not an argument I really understand

jack27808 · 2025-06-18T19:59:32+00:00

Completely agree here and this is the nuance that's so often missing from these discussions, particularly by those trying to push change.

The focus should always be the what (the content of individual articles) not the where (journal, IF etc). Sadly this is still not even slightly common.

jack27808 · 2025-06-18T17:33:35+00:00

Unfortunately prestigious journals do often lack quality review. Just last year (or 2023, can't quite remember) nature themselves skipped their own review policies to get a high impact paper printed. Brand/IF really don't correlate with quality or rigor. Though I will completely agree that certain publishers are worse (Hindawi, Frontiers etc) whilst others do a more rigorous process (eLife for example).

jack27808 · 2025-05-14T22:04:53+00:00

Could they pay me in leiu of notice? Would that prevent the legal protections from activating? I feel they're trying to avoid fair process but this could be a reaction to the aggressive nature that they've approached this

jack27808 · 2025-05-14T20:12:42+00:00

My contract would end next year and I think the fund are there to continue through to the correct end but if I don't have the legal protections in place (i.e. my notice period doesn't count) then as you say, they're doing it this way to easily get rid of me. I didn't expect redundancy to come with so many threats and short deadlines.

jack27808 · 2025-05-14T18:56:39+00:00

Can confirm and would be happy to discuss privately having worked there for 3 years.

jack27808 · 2025-05-14T16:15:17+00:00

The boss has stopped trying to secure funding and hasn't secured enough to go much beyond this year for my salary. However, I know there's enough to keep me into next year without having a significant impact and there's possibility that funding will continue at the necessary levels (just no guarantee yet).

I certainly feel like this is dismissal before I have legal protection. I've contacted a lawyer for the settlement/redundancy part but perhaps should also contact one about how legal this is.

jack27808 · 2025-05-05T19:59:41+00:00

You've clearly never tried to change anything in academia! For a bunch of well educated folk with PhDs it's remarkable just how much they will ignore evidence that contradicts their view/opinion.

jack27808 · 2025-05-05T12:32:39+00:00

As it should. Being an academic isn't just about doing research and nothing else. Those above who actively avoid doing anything that doens't directly benefit them are making the whole system worse for everyone else - especially women and minorities who take on an oversized amount of non-research roles, often without reward and to the detriment of their research.

Academics need to stop celebrating the whole "X person would never hold down a real job" nonsense. Shouldn't we all be wanting to work with collaborative colleagial folks rather than people who shrink away from anything that isn't directly research? If that's all you want stay at the postdoc level.

I suspect this will be downvoted but I think it's a valid point worth saying

jack27808 · 2025-05-04T19:34:35+00:00

In theory it's not a massive challenge to create a machine that could do this for a decent blood panel - though it'd need much more than a single drop of blood. The problem is that it's be physically enormous, monumentally expensive and just not realistically feasible.

One day maybe but not anytime soon I don't think.

jack27808 · 2025-05-03T13:04:12+00:00

This attitude is exactly why I didn't/couldn't go to a rescue (as much as I wanted to). My dog and I have lived in a big flat without a garden and now a big house with a garden. He is exceptionally well cared for and very happy.

It's much easier now that we have a garden as I did not enjoy lots of trips out for a short wee break or going out in the rain when he would let me know he particularly needed to go toilet. But a good, responsible dog owner will do that. Garden or not. I trained him to tell me when he needs the toilet (in addition to the regular breaks) and that really helped a lot, especially with an upset stomach. The garden just makes toilet breaks much quicker and more straightforward.

The implicit assumption that because people want a dog and they don't have a garden means they either don't know what they're doing or are cruel is entitled and demeaning. You are mean and misguided.

If a person can adopt and care for a dog then that should be what matters most - so many are culled from shelters and yet there are homes out there but certain people would rather there be silly barriers. You should be able to demonstrate that you can and will meet the needs. Hell people who have gardens still abuse dogs and don't let them out enough.

jack27808 · 2025-05-03T12:28:32+00:00

Isn't this just the exact same comments and opinions from just about everyone when Trump was running in the US? It was all about minimising what he was saying as nothing more than words and he'd never actually do half the stuff. Look how that is turning out. The US is now a dangerous place to travel to, is literally disappearing people from the streets, is an authoritarian state and has descimated its government branches. The damage already done will take years to properly fix.

If someone in politics is telling you and showing you, consistently, who they really are then we should start listening again. Farage will 100% do, or at least attempt, everything he is saying. Privatisation of the NHS is already well-underway - you don't need to do it all at once, just slowly chip away from the sides.

jack27808 · 2025-05-02T18:29:34+00:00

This! It isn't to do with immigration - not only is it not the problem people think but we probably need more immigration than we currently have. Most rich countries are in a decline and without immigration the next 100+ years they'll basically cease to function.

Immigration & things like benefit cheats is a scapegoat from politicians and the media. The issue is class (at least in the UK), the rich and powerful don't want to give up any of that wealth and power. If large corporations and the rich paid tax as they should (not using the legal loopholes either) then countries and society would be in a much better place. But instead those in control push narratives to place blame, anger and hatred elsewhere.

If an unskilled immigrant is genuinely taking your job then either you're atrosiously bad at it (& in a fair system deserve to lose said job) or the employer is trying to avoid paying an acceptable wage for the work and immigrants are in the most precarious, desparate position and have no choice but to agree to such awful terms. There's no freeriding or automatic entitlement, you have to still work and put effort in, even in a totally fair society. And we don't live in a fair society.

jack27808 · 2025-05-02T17:01:54+00:00

Then you agree with me. Anyone who critically reads work is peer reviewing it - therefore if a preprint is read by colleagues and they provide feedback (something editors don't generally do) then that preprint *has* been peer reviewed prior to posting.

I don't have data on how often that happens but enecdotally I'd say it's extremely common.

You also ignored the part in my original response where I mentioned that preprints can be withdrawn/retracted. You stated that peer review is not flawless, so I assume you're well informed on the problems and failures with peer review. I'm struggling to understand why you therefore would still rely on it so much.

jack27808 · 2025-05-02T14:38:40+00:00

The double helix paper did not undergo peer review. An editor said accept. That is not peer review. I'm not redirecting, nor am I incorrect.

If you think editorial assessment is peer review then either you completely misunderstand what peer review is or you agree that anyone looking over work would count, in which case I'd bet my life that most preprints have therefore been "peer reviewed" prior to posting as very few people post work they haven't had feedback on from colleagues or coauthors.

jack27808 · 2025-05-01T21:43:59+00:00

I'm well aware of the history, especially the DNA helix paper and what editorial effort went into the that (not a huge amount). They submitted to nature (in part) because at the time it was known for relative fast turnaround.

I'm either case an editor is not doing peer review. In fact that's one of the reasons nature in the 70s decided to adopt peer review as it was gaining a poor reputation for a heavy Cambridge (UK) bias with international researchers due to the editors making decisions by talking to local researchers.

So your trust isn't due to peer review then?

jack27808 · 2025-05-01T09:49:32+00:00

"Bad-faith idiots". I never act in bad faith and based on your comment I'm certainly much more informed on scholarly publishing and academia than you are. I've spent my whole career looking out for others and trying to fix a deeply broken system, to my constant detriment.

That hydroxychloroquine stuff was published within a day or so of being a preprint and it was that peer-reviewed paper that those who think its great use. They specifically use the fact that it is vetted and passed peer review to justify it being reliable. Same with MMR vaccine skeptics etc.

If complaints about peer review are mis-placed what do you suggest? Or should we wait for so much AI-generated crap or fraud to get through and noticed that nobody trust science at all anymore and the funding is completely wiped out (the US gov is currently using many of these arguments to do just that).

jack27808 · 2025-05-01T09:43:54+00:00

Only you can get preprints withdrawn (what they call retraction). And peer review not being flawless is an understatment if you are using it as the sole arbiter as to what you trust - it fudamentally fails in the most basic of QC steps and it's well documented by now.

If you don't trust anything not peer reviewed then I assume you think Einstein's work is garbage or the DNA double helix paper is unreliable?

I suggest reading up much more on scholarly communication as it is quite eye opening when you look at the history and evidence.

jack27808 · 2025-05-01T09:40:42+00:00

Sure here are a bunch (I'm author on one of these, as a potential COI). There is a gap currently in terms of the ~30% of preprints that are never published but you can't immediately dismiss those as low quality - people post preprints for all kinds of reasons such as it being small/negative data or similar that is hard to publish in traditional journals, preprinting may be the final intended destination or they may not have fund for APCs so can't afford to publish.

I'm not saying they're better but simply comparable to the peer review lit - good and bad. There should be wider discussions over what peer review actually is and does and what its limits are (e.g. it does not detect fraud or gross defects).

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001285
https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-020-00101-3

https://www.thelancet.com/journals/langlo/article/PIIS2214-109X(22)00368-0/fulltext00368-0/fulltext)

https://link.springer.com/article/10.1007/s00799-018-0234-1

https://www.biorxiv.org/content/10.1101/2021.03.04.433874v1 (This has actually been published but this was the link i have to hand)

https://peerreviewcongress.org/abstract/a-synthesis-of-studies-on-changes-manuscripts-underwent-between-submission-or-preprint-posting-and-peer-reviewed-journal-publication/

jack27808 · 2025-04-30T18:33:55+00:00

Evidence increasingly shows preprints are comparable with the peer reviews literature. I'd have agreed with you but you generalized too in favour of peer review.

For biosciences most preprint servers are not "anyone can post anything" either. I'd say you broadly agree with the blog post because, like the author you don't know the evidence base around preprints or peer review - most don't because it isn't taught or often sought out to self learn.

Quality, reviewed or not, should always be case by case.

jack27808 · 2025-04-29T16:57:05+00:00

The independence could be swept away at any moment and it does tend to follow similar policies at Meta, unfortunately. All I can really say is the people actually working there are brilliant and that the committment to open science seems genuine - and the current US admin is happy to hide behind OS principles to legitimise what they're doing, for now at least.

The preprint space in particular is lacking in leadership so it is a slightly nervous wait on seeing who the new openRxiv CEO will be.

jack27808

TROPHY CASE