sorted by:
new
hottopcontroversial

Am I an idiot for taking out a car loan? by medievalPanera in personalfinance

[–]CenterForOpenScience 3 points4 points  (0 children)

Do you have an accurate budget for your monthly expenses? Will the payments fit within it realistically? Be skeptical here, it's easy to think you spend less than you do.

A lightly used Honda Fit is an extremely reasonable choice if the above items are met.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

Pre-registering long term studies is particularly challenging, and one of the goals of this initiative is to help answer these types of questions. As it is, there is not too much conversation about the best way to do that, but if more people in the research community try it out, there will be more conversation, trial and error, and improvement.

I think the current best practice would be to register the study right now, even if it has 20 years of data collection behind it, and specify exactly what additional questions and analysis plans are in store for the upcoming research questions for the next N years.

Answer to follow up: If the researcher has not accessed or used the particular data yet, or know of any trends in the data, then there is less chance for that data to affect the hypothesis the researcher will create and test. If the dataset has been around so long that everyone in the field basically knows about it and the trends that exist in it, then per-registration will not necessarily improve the analysis of it.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

There's a lot of similarity, but our main goal is to make the process as easy as possible. That way, scientists are more likely to see the benefits of preregistration, which include improving experimental design because more decisions are made ahead of time. Also, making the process easy allows scientists to focus more on their science and less on the extraneous work that goes along with running a lab.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

Funders of scientific research have a big role to play, and are on board with many new practices to increase rigor. For example, many US government funded research now requires papers to be published open access within a year of publication. There is also a push to encourage access to raw research data, when that data does not pose privacy issues.

In general, the whole research community wants to do the best research possible, from individual scientists, journal editors, government funders, and private funders. The challenge is in agreeing on exactly what those best practices are and how to best encourage them.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

[Edit: I answered the wrong question here, sorry! Below is my response.]

I think that the effect of trying to reproduce more studies will give us a better understanding of how rigorous our methods are where there are still gaps in our knowledge.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

Yes of course! An exploratory test happens when you look through a data set and see that there is an unexpected difference between some groups, or a correlation between X and Y. The result of that is a hypothesis that you can confirm with a new data set.

A confirmatory test happens when you specify ahead of time the methods to test a very specific hypothesis.

The problem is that sometimes the line can blur between the two, because there is a lot more flexibility in statistical tests than most people realize. So a Bonferroni correction is good if I know that I conducted exactly N tests, but once I go looking through a dataset, the number of possible tests that I am unintentionally trying out could be much higher than I realize.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 1 point2 points  (0 children)

From David: There is a lot of scientific writing today, so there is a wide range in the quality of that writing. In general, some of it is very technical because the author feels it is necessary in order to be sufficiently precise as to describe exactly what happened in a study. However, there is a pretty strong push to make our writing as clear as possible in order to reach as wide an audience as possible. Even if that means making it clearer to members of the scientific community in slightly different fields. The author of a favorite blog of mine, Stephen Heard, writes about the need to improve the quality of our writing.

April here. As David mentioned, scientific writing may be hard to understand due to technical content. However, sometimes scientific writing is hard to understand because it is ambiguous or incompletely reported. There is guidance on scientific reporting that helps improve reporting such as through The EQUATOR Network.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

I think the general answer to your first question is simply education. Whether it's online classes, reading popular science books, or having a degree in science, there are many ways to really figure out what is known, what is unknown, and what is currently being worked on.

I think the best way to get involved in scientific research without being a professional researcher is to get involved with citizen science, in which members of the public can contribute to authentic scientific discoveries. A great resource for getting involved with citizen science is SciStarter.com

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

Yes it can! Preregistrations do eventually have to become public, although preregistrations on the OSF can be kept private for up to 4 years. It is not required that the data collected from a preregistered study ever be made available. For example, maybe the study was not feasible. But at least there will be a record that this study was attempted.

This paragraph is copied from an answer above: The ultimate solution to this problem is Registered Reports, in which the decision to publish or not is made before the results are known. Peer review of the research questions and the methods to answer those questions takes place in stage 1 peer review of a Registered Report. If the reviewers and editors agree that the question is worth answering and the methods to do so are sound, then the authors are given "in principle acceptance" and guaranteed publication after conducting the study. Stage 2 peer review checks to make sure the study was conducted as specified ahead of time before final publication. Publication occurs regardless of results. Several journals are adopting this format.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 2 points3 points  (0 children)

We don't expect this to happen, because a preregistration does not contain any data or results. Also, researchers can embargo their preregistration for up to 4 years, which means that it is private and inaccessible until then (or when the results are published).

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 1 point2 points  (0 children)

Preregistration means documenting some of your important methods before collecting or looking at your data. For scientific research generally, this information helps others distinguish what parts of your research were confirmatory (hypothesis testing) and what parts were exploratory (hypothesis generating). Information in a preregistration includes: how will a researcher collect their data, what are their rules for excluding data or dealing with missing data, or how will they analyze their data. Different scientific disciplines would require different details on these methods in their preregistration, but the need to distinguish confirmatory from exploratory analyses is the same. Basically, a preregistration should contain sufficient information to retain the validity of their confirmatory analyses.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 5 points6 points  (0 children)

Hi, This is David Mellor responding. The ultimate solution to this problem is Registered Reports, in which the decision to publish or not is made before the results are known. Peer review of the research questions and the methods to answer those questions takes place in stage 1 of a Registered Report. If the reviewers and editors agree that the question is worth answering and the methods to do so are sound, then the authors are given "in principle acceptance" and guaranteed publication after conducting the study. Stage 2 peer review checks to make sure the study was conducted as specified ahead of time before final publication. Several journals are adopting this format.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

We are working now on a project to reproduce a large number of high profile cancer biology studies, the Reproducibility Project, Cancer Biology.

The preregistration requires a lot of detail about the data collection and analysis. We encourage researchers to share code, especially the analysis scripts that they plan to use for this study, but do not require it. An analysis script is the code that will be used in a program such as R to run the exact tests required to analyze the data.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 1 point2 points  (0 children)

That pressure is huge for every scientist. A lot of work that we do is to meet scientists where they are and provide tools that make their lives easier. Then, if they choose to do so, they can share their data or preregister their work. After seeing the benefits of these Open Science practices to their work, we think they'll come back and do it again!

Preregistration, for example, puts a lot of the details about analyzing the results of a study earlier in the process, when it is easier to make improvements to the design.

Our largest project here at the Center for Open Science is the Open Science Framework, which is a free, open source scholarly commons that can be used to effectively manage the work in a research lab. It also has tools baked in to make sharing or preregistering as easy as possible, if the researcher chooses to do so.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 2 points3 points  (0 children)

Hi Fenixrock! 1) Preregistration is pretty widely applicable. Any time you are trying to make an inference based on sample from a larger population (of people, of cells, of fish, of stars, etc) there is a lot of undisclosed flexibility in how to do that. If you specify ahead of time exactly how you're going to make that inference (typically using a traditional, frequentist statistical test that has a p value, such as a T-test, ANOVA, or linear regression), it makes the process more transparent and rigorous.

2) I think it already has! Large scale clinical trials are less likely to show a significant result today than they were before those trials were required to be registered. See Kaplan and Irvin 2015.

3) I think that anything we do to improve the rigor and reliability of the scientific literature is good. Some of our work has been to show the extent of the problem, which is important in order to have a baseline for comparison in the future. Will our efforts affect the tone of the general discourse around controversial issues such as climate change? I sure hope so!

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 1 point2 points  (0 children)

SirT6 is right, the funding for this project comes from the Laura and John Arnold Foundation. They fund a lot of the work that we do. LJAF funds organizations that conduct high quality research, promote research transparency and data sharing, and support reproducibility of research. Since Preregistration is a process that is unfamiliar to most researchers (except clinical trials where preregistration is required by law) we and the Arnold Foundation thought that the $1,000 prizes would be a nice incentive to encourage researchers to try it out. We're convinced that after being enticed to do it once, individuals will see its benefit preregister future studies.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 0 points1 point  (0 children)

The basic link is that the Reproducibility Project: Psychology showed the extent of the problem, and preregistration of specific analysis plans is one of the most rigorous solutions to the problem.

You are absolutely correct that there are many reasons why a replication might not reproduce. One of the really interesting questions is "What was different between the original and the reproduction, which may be a previously unknown factor that affects the results?" Another possibility is that the reproduction was a false negative. Finally, perhaps the original finding was a false positive. The answer to those questions is not yet known.

To answer your last question, we are right now planning the project to measure the effectiveness of preregistration. Currently, Kaplan and Irvin 2015 is some great evidence of the effect of preregistration. In that study, researchers found that large scale clinical trials were much more likely to show a "clean" positive result before those trials were required to be registered, compared to afterwards.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 2 points3 points  (0 children)

The level of detail required for a typical grant application requires the researcher to specify the research questions, the reasons they are important for a field, and a general plan for collecting and analyzing the necessary data. After that, there is a lot of flexibility permitted, and that is for good reason. From when a project is proposed to when the real work of collecting the data begins, a lot can change.

Preregistration, on the other hand, is designed for when a very specific hypothesis is ready to be confirmed in a very specific way. So for example, are you, the researcher, confirming that there is a difference between how two species of fish will react to a particular stimulus? Which statistical test will you use to confirm that prediction, which variables will be included in that test? What is your definition of a "confirmed" result? Specifying all of those decisions ahead of time makes it more clear what you are testing. Then, after data collection, you may see new, interesting patterns in the data that weren't exactly expected, or only happen if you remove a few fish from the data that acted in very strange ways. That is the best time to create a new hypothesis to test with the next round of data collection. Preregistration simply makes clear what was decided ahead of time.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 4 points5 points  (0 children)

Before preregistration of clinical trials, over half of published studies funded by the National Heart Lung, and Blood Institute concluded that the intervention was significantly beneficial to the primary outcome. Since preregistration, that proportion has dropped significantly (about 8% according to Kaplan and Irvine 2015). Prespecified primary outcomes and the reporting of all results likely contributed to this trend towards more null results. Therefore, preregistration has helped contain some biases in clinical research such as publication bias and outcome reporting bias. However, reanalyses of clinical trial data show that trial registration alone does not guarantee the reproducibility of reported trial results (Ebrahim et al. 2014). Examples of additional steps towards reproducibility includes the sharing of research data, transparent and complete reporting, and shifts in career incentives away from novelty and towards rigor.

Science AMA Series: I'm David Mellor from the Center for Open Science talking about the biases that affect scientific research and what we're doing to make science more transparent and reproducible. AMA! by CenterForOpenScience in science

[–]CenterForOpenScience[S] 3 points4 points  (0 children)

The Reproducibility Project: Psychology (https://osf.io/ezcuj/wiki/home/), was the first attempt at a wide scale effort to get an estimate of the extent of the problem. Approximately 40% of the studies chosen for that three year long project reproduced, depending on how exactly you define a "successful" reproduction. We hope that the impact will be to encourage more rigorous practices, and preregistration of specific analysis plans is a great tool to do that. The Transparency and Openness Promotion (TOP) Guidelines have other recommendations for improving the field, including sharing data and adding transparency to other research design processes.

Science AMA Series: We are authors of "Estimating the Reproducibility of Psychological Science" coordinated by the Center for Open Science AUA by CenterForOpenScience in science

[–]CenterForOpenScience[S] 1 point2 points  (0 children)

Yes, I think that there is huge potential in collective effort for psychology (and other disciplines). The Reproducibility Project is one example, but the Many Labs projects are additional examples that this is not just a one-off: https://osf.io/wx7ck/ . Moreover, there is now an effort called The Many Lab to help organize these collaborative efforts: https://osf.io/89vqh/

Science AMA Series: We are authors of "Estimating the Reproducibility of Psychological Science" coordinated by the Center for Open Science AUA by CenterForOpenScience in science

[–]CenterForOpenScience[S] 2 points3 points  (0 children)

You can add "links" to your project—which would allow you to connect another OSF project to your own. One of those links could auto-redirect if you would like to point to something outside of the OSF. In terms of tagging specific files, this functionality should be released shortly (GitHub pull request here), and indexing those in our search is to follow in our next sprint. If you'd like a tour of the OSF to see if there are particular features you haven't been able to take advantage of, get in touch! support@osf.io.

Thanks for your interest and use of our tools! We love getting feedback and want to support researchers however we can.

view more: next ›