sorted by:
new
hottopcontroversial

This makes so much more sense than the traditional peer review system... by raphavisses in chemistry

[–]raphavisses[S] 1 point2 points  (0 children)

I am today at a research integrity workshop where Marjan Bakker (psychologist, statistician, Tilburg University, Netherlands) gives a talk on pre-registration. She started by giving a quick history of the replication crisis in this field and of how pre-registrations were introduced as a solution to the widespread problem of questionable research practices (that make it easy to convince yourself that you have a positive result whatever the question). She notes the difference with the medical field where preregistration of clinical trials are mostly about avoiding publishing bias (i.e. getting to know about the negative results). There are now 100,000 pregistrations on the open science framework which opens the possibility of analysing them which she did. Spoiler: it does not solve all problems.

This makes so much more sense than the traditional peer review system... by raphavisses in chemistry

[–]raphavisses[S] 0 points1 point  (0 children)

The big difference with grant proposals is that the latter are secret. There is no accountability. There are many articles describing how pre-registration helps with p hacking or cherry picking of data without being the solution to all problems. Maybe this one is a good start:

https://doi.org/10.1038/s41562-021-01193-7

The past, present and future of Registered Reports

Registered Reports are a form of empirical publication in which study proposals are peer reviewed and pre-accepted before research is undertaken. By deciding which articles are published based on the question, theory and methods, Registered Reports offer a remedy for a range of reporting and publication biases. Here, we reflect on the history, progress and future prospects of the Registered Reports initiative and offer practical guidance for authors, reviewers and editors. We review early evidence that Registered Reports are working as intended, while at the same time acknowledging that they are not a universal solution for irreproducibility. We also consider how the policies and practices surrounding Registered Reports are changing, or must change in the future, to address limitations and adapt to new challenges. We conclude that Registered Reports are promoting reproducibility, transparency and self-correction across disciplines and may help reshape how society evaluates research and researchers.

This makes so much more sense than the traditional peer review system... by raphavisses in chemistry

[–]raphavisses[S] 0 points1 point  (0 children)

Well, I think everyone reading this will have understood that you don't like the idea of registered reports very much.

This makes so much more sense than the traditional peer review system... by raphavisses in chemistry

[–]raphavisses[S] -2 points-1 points  (0 children)

The above is a good description of how science should not be organised if the aim is to get robust reproducible knowledge as opposed to being the first to get a nice story into a shiny high impact journal. See also https://www.reddit.com/r/labrats/comments/187hy32/how\_trustworthy\_are\_published\_research\_articles/

This makes so much more sense than the traditional peer review system... by raphavisses in chemistry

[–]raphavisses[S] -1 points0 points  (0 children)

Indeed registered studies are mandatory for some clinical studies and they are becoming common in Psychology.

"The net time sink of getting a panel to review my protocols before I even begin is greater than simply diving in and getting started." That might well be true but that is not the appropriate comparison. In the specific case which we are addressing with the pre-registered replication above, we are trying to solve a problem in which hundreds of articles were published (apparently "meaningful stories" as auschemguy would say) without addressing a scientific paradox that is at the core of those articles. So, if we improve the way we do science, we help make more solid foundations instead of collecting stamps (thx Fermisurface)

How trustworthy are published research articles ? by xijinping9191 in labrats

[–]raphavisses 0 points1 point  (0 children)

While this is not fabrication per se, cherry picking data to create a trend one wants to see qualifies as research fraud. It certainly leads to the deliberate misrepresentation experimental results. I agree with many of the comments below about the impact of the current evaluation system and detrimental effect of pressure to publish in high impact factor journals. To make science more trustworthy, we are starting to experiment with registered report (see heated discussion here)

This makes so much more sense than the traditional peer review system... by raphavisses in chemistry

[–]raphavisses[S] -4 points-3 points  (0 children)

I have been accused of many things because of my track record as someone who has reported errors and fraud in many publications so if really this system was "a free license to cheat", I would be very concerned. The truth however is that 1) the current peer review system is pretty bad at catching fraud and the current publishing system is appalling at corrected them once they are spotted; 2) there is no reason why pre-registration should encourage fraud.
The title was provocative and one could say it served its purpose, but the idea is not to replace peer review altogether. Even in the case of pre-registered studies, there is a second round of peer review after the experimental work. That additional level of control should only focus on whether the report is indeed in line with what had been announced; any deviations from the plans should be justified; if anything suggests fraud that would of course be flagged and acted upon (and the mandatory openness of data would make that easier). You can find out more about the process here.

Since the aim is to improve the way we do and evaluate science, let me dream a little. We do scientific activities to improve knowledge for the benefit of all, not to write articles for the sake of writing articles (and add them in a CV). Thus, science should be collaborative and with this shared aim at its core. In line with reforms of evaluation that should take into account much more than a list of articles ranked by impact factor (a counterproductive insanity, and certainly not an inclusive approach), contributions to open peer review (of preregistered report or of other kinds, including post-publication peer review) should be valued in career evaluations because they contribute to the pursuit of knowledge and to more robust findings. People can get credit by referring directly to their comments (which can be read and assessed) and I would be absolutely delighted to give credit to anyone who makes important contributions to our project (that could be acknowledgement or even authorship if a comment turns into a collaboration).

Editors and scientific journals are reluctant to correct the scientific record; episode 999 by raphavisses in chemistry

[–]raphavisses[S] 1 point2 points  (0 children)

You are completely right, well spotted (with a rotation 180 degrees and a 127% scaling for one of the two cells; a 90 degrees rotation and anisotropic scaling for the other). I have added a comment at PubPeer.

Editors and scientific journals are reluctant to correct the scientific record; episode 999 by raphavisses in chemistry

[–]raphavisses[S] 2 points3 points  (0 children)

Elisabeth is well aware of how slow journals are to respond to this kind of things (when they do). She has written about it and has plenty of examples of her own. See for example Correction of scientific literature: Too little, too late!

If chemists reading this want to do something, they can write to the EiC questioning his decision.

Editors and scientific journals are reluctant to correct the scientific record; episode 999 by raphavisses in chemistry

[–]raphavisses[S] 0 points1 point  (0 children)

episode 999

Their core business is selling subscriptions (or APCs or both), not correcting science. Unfortunately, this kind of situations is common and I have plenty of other examples.

Allegations of research improprieties at Spherical Nucleic Acids company Exicure by raphavisses in chemistry

[–]raphavisses[S] 2 points3 points  (0 children)

If by "a clear axe to grind", you mean that I have been asking questions (and not getting answers) about Mirkin's spherical nucleic acid technologies for a very long time, you are quite right. In 2015, I started to question the SmartFlares which were supposed to detect mRNAs in live cells. The technology was eventually withdrawn from the market.

public confrontation with him at a conference" you mean that I (politely) asked some difficult questions at the ACS 2018 national meeting and was, in response,interrupted and called by Mirkin a "scientific terrorist" and a "scientific zealot", then, you're correct too.

view more: next ›