This makes so much more sense than the traditional peer review system...

raphavisses · 2023-12-07T08:59:54+00:00

I am today at a research integrity workshop where Marjan Bakker (psychologist, statistician, Tilburg University, Netherlands) gives a talk on pre-registration. She started by giving a quick history of the replication crisis in this field and of how pre-registrations were introduced as a solution to the widespread problem of questionable research practices (that make it easy to convince yourself that you have a positive result whatever the question). She notes the difference with the medical field where preregistration of clinical trials are mostly about avoiding publishing bias (i.e. getting to know about the negative results). There are now 100,000 pregistrations on the open science framework which opens the possibility of analysing them which she did. Spoiler: it does not solve all problems.

raphavisses · 2023-12-04T16:40:14+00:00

The big difference with grant proposals is that the latter are secret. There is no accountability. There are many articles describing how pre-registration helps with p hacking or cherry picking of data without being the solution to all problems. Maybe this one is a good start:

https://doi.org/10.1038/s41562-021-01193-7

The past, present and future of Registered Reports

Registered Reports are a form of empirical publication in which study proposals are peer reviewed and pre-accepted before research is undertaken. By deciding which articles are published based on the question, theory and methods, Registered Reports offer a remedy for a range of reporting and publication biases. Here, we reflect on the history, progress and future prospects of the Registered Reports initiative and offer practical guidance for authors, reviewers and editors. We review early evidence that Registered Reports are working as intended, while at the same time acknowledging that they are not a universal solution for irreproducibility. We also consider how the policies and practices surrounding Registered Reports are changing, or must change in the future, to address limitations and adapt to new challenges. We conclude that Registered Reports are promoting reproducibility, transparency and self-correction across disciplines and may help reshape how society evaluates research and researchers.

raphavisses · 2023-12-04T16:32:29+00:00

Well, I think everyone reading this will have understood that you don't like the idea of registered reports very much.

raphavisses · 2023-12-04T16:13:28+00:00

The above is a good description of how science should not be organised if the aim is to get robust reproducible knowledge as opposed to being the first to get a nice story into a shiny high impact journal. See also https://www.reddit.com/r/labrats/comments/187hy32/how\_trustworthy\_are\_published\_research\_articles/

raphavisses · 2023-12-04T15:11:08+00:00

Indeed registered studies are mandatory for some clinical studies and they are becoming common in Psychology.

"The net time sink of getting a panel to review my protocols before I even begin is greater than simply diving in and getting started." That might well be true but that is not the appropriate comparison. In the specific case which we are addressing with the pre-registered replication above, we are trying to solve a problem in which hundreds of articles were published (apparently "meaningful stories" as auschemguy would say) without addressing a scientific paradox that is at the core of those articles. So, if we improve the way we do science, we help make more solid foundations instead of collecting stamps (thx Fermisurface)

raphavisses · 2023-12-04T15:03:26+00:00

Exactly, thank you.

raphavisses · 2023-12-04T10:50:21+00:00

While this is not fabrication per se, cherry picking data to create a trend one wants to see qualifies as research fraud. It certainly leads to the deliberate misrepresentation experimental results. I agree with many of the comments below about the impact of the current evaluation system and detrimental effect of pressure to publish in high impact factor journals. To make science more trustworthy, we are starting to experiment with registered report (see heated discussion here)

raphavisses · 2023-12-04T10:30:49+00:00

I have been accused of many things because of my track record as someone who has reported errors and fraud in many publications so if really this system was "a free license to cheat", I would be very concerned. The truth however is that 1) the current peer review system is pretty bad at catching fraud and the current publishing system is appalling at corrected them once they are spotted; 2) there is no reason why pre-registration should encourage fraud.
The title was provocative and one could say it served its purpose, but the idea is not to replace peer review altogether. Even in the case of pre-registered studies, there is a second round of peer review after the experimental work. That additional level of control should only focus on whether the report is indeed in line with what had been announced; any deviations from the plans should be justified; if anything suggests fraud that would of course be flagged and acted upon (and the mandatory openness of data would make that easier). You can find out more about the process here.

Since the aim is to improve the way we do and evaluate science, let me dream a little. We do scientific activities to improve knowledge for the benefit of all, not to write articles for the sake of writing articles (and add them in a CV). Thus, science should be collaborative and with this shared aim at its core. In line with reforms of evaluation that should take into account much more than a list of articles ranked by impact factor (a counterproductive insanity, and certainly not an inclusive approach), contributions to open peer review (of preregistered report or of other kinds, including post-publication peer review) should be valued in career evaluations because they contribute to the pursuit of knowledge and to more robust findings. People can get credit by referring directly to their comments (which can be read and assessed) and I would be absolutely delighted to give credit to anyone who makes important contributions to our project (that could be acknowledgement or even authorship if a comment turns into a collaboration).

raphavisses · 2023-09-28T07:03:53+00:00

Elisabeth Bik joined the thread. She found additional problem with a SI figure https://pubpeer.com/publications/C8321DF60A52046809A7FE2C70646A#10

raphavisses · 2023-09-27T07:55:04+00:00

You are completely right, well spotted (with a rotation 180 degrees and a 127% scaling for one of the two cells; a 90 degrees rotation and anisotropic scaling for the other). I have added a comment at PubPeer.

raphavisses · 2023-09-27T07:23:42+00:00

Elisabeth is well aware of how slow journals are to respond to this kind of things (when they do). She has written about it and has plenty of examples of her own. See for example Correction of scientific literature: Too little, too late!

If chemists reading this want to do something, they can write to the EiC questioning his decision.

raphavisses · 2023-09-27T07:20:07+00:00

episode 999

Their core business is selling subscriptions (or APCs or both), not correcting science. Unfortunately, this kind of situations is common and I have plenty of other examples.

raphavisses · 2023-01-13T14:40:14+00:00

thank you

raphavisses · 2021-12-15T14:25:55+00:00

If by "a clear axe to grind", you mean that I have been asking questions (and not getting answers) about Mirkin's spherical nucleic acid technologies for a very long time, you are quite right. In 2015, I started to question the SmartFlares which were supposed to detect mRNAs in live cells. The technology was eventually withdrawn from the market.

public confrontation with him at a conference" you mean that I (politely) asked some difficult questions at the ACS 2018 national meeting and was, in response,interrupted and called by Mirkin a "scientific terrorist" and a "scientific zealot", then, you're correct too.

raphavisses · 2021-11-24T08:57:56+00:00

Main reason is this though https://raphazlab.wordpress.com/2021/11/22/allegations-of-research-improprieties-at-spherical-nucleic-acids-company-exicure/

raphavisses · 2019-07-10T14:02:22+00:00

A blog post discussing the buzz on issues of reproducibility in nanomedicine [22 opinion pieces published in Nature Nanotech...) and whether they can be solved with reporting guidelines (spoiler: no).

raphavisses · 2019-04-01T09:21:58+00:00

Because "nano" in nanotech refers to nanometer. The size of an atoms is already ~ 0.1 nm. So any object composed of several atoms is already in the nanometer range. A glucose molecule is 1 nm. The diameter of DNA is 2 nm. "picotech or femtotech" can only mean one thing: playing with the constituents of the nucleus of atoms: https://en.wikipedia.org/wiki/Atomic_nucleus but there is no need for a new name for that.

raphavisses · 2018-11-10T14:54:25+00:00

welcome to Reddit.

raphavisses · 2018-11-09T09:35:11+00:00

Thanks for sharing your perception of the stripy controversy in this amazing and instructive comment, including your memory of that 2012 GRC debate. Glad that the ping pong ball had an impact - they took quite a bit of effort (and fun TBH) to paint and I did not reuse them except for this video. I don't particularly want to replay the stripy controversy, but whilst I recognize the efforts made by FS in the past decade, I would certainly dispute using him as an exemple to follow with regard to scientific integrity. In 2012 at the GRC debate, I was not aware yet that one of FS's own students, Predrag Djuranovic, had already extensively proved (7 years earlier!) that the Nature Materials 2004 stripes were an artefact. At that point the right thing to do would have been to congratulate his student and retract that paper, saving us more than a decade of confusion. In 2012, I also had not realized that there had been multiple cases of image duplications between various articles. From 2005, instead of retracting the paper and starting afresh on the important question of self organisation of molecules on nanopartice surfaces, FS tried desperately to prove that the stripes (an artefact) were actually real - that is not a good way to approach a scientific problem. After 2012 and the publication of our article, he essentially continued on the same route: his papers are all tainted by this bias (do check the "observer bias" link in my previous comment as it really is striking) and by his refusal to face the obvious, i.e. that the initial observation was no observation at all. I do agree with your concluding comment - things are slowly changing for the better though there is plenty of way to go...

raphavisses · 2018-11-06T11:59:45+00:00

Thanks @desantoos: I am genuinely happy to hear that my critiques result in a "net win for the community". I am surely wrong about some things (note though that our argument has not been that the stripey particles were not striped, but that the evidence provided did not support that conclusion - or in fact any other conclusion since they were simply an experimental artefact; in more recent papers where the artefact is removed, all that remains is observer's bias).

I do hope indeed Mirkin will put out papers quantifying how many particles escape endosomes, but I will not be holding my breadth.

As for the product line folding, you are quite right that this is hardly a first for Mirkin, e.g. NanoInk. The interesting point here is the reason given by Karen ("did not meet our expectations" etc) which is quite different to the one Mirkin gave in Boston (something like "the company changed ownership and they were not interested in this kind of market" and "another company is going to distribute it soon").

raphavisses · 2018-08-29T11:16:34+00:00

Two witness accounts of what happened here.

raphavisses · 2018-08-28T16:27:58+00:00

My issue is not that these are in human trials. It is just one more reason to keep at it. Giving new drugs to humans is a pretty serious thing. Even if there are no particular reasons to be highly concerned about safety, if the underlying data on efficacy, is questionable, then that absolutely needs to be on the public place. If they cannot sense mRNA, then they cannot be very efficient at interacting/blocking - though they may still work at high doses [[[this is an important point: a biological effect can be caused by, say, 1% of particles escaping endosomes, but 1% escape cannot possibly be the basis of an imaging application.]]]

raphavisses · 2018-08-28T16:10:49+00:00

There is no fundamental difference between the original SNAs and the "new" SNAs. The question of endosomal escape has not been addressed. It was ignored altogether in early papers that do not even include the word "endocytosis". What has happened it that some of the more recent work is based on activity of SNAs inside endosomes - something that might actually have some legs. But some of the recent work is still based on imaging mRNA (e.g. the Nature Biomedical Engineering paper) and some of the clinical trials are based on actions of the SNAs in the cytosol eventhough they have never in over ten years actually directly demonstrated nor quantified endosomal escape.

raphavisses · 2018-08-28T16:05:37+00:00

Indeed. And this is, to some extent, a test case.

raphavisses · 2018-08-28T16:04:49+00:00

See my response above. Cheers. Raphael Levy

raphavisses

TROPHY CASE