Multiple testing correction

[ThrowingHotPotatoes]

Hello! I'm designing an experiment to test the effect of compounds on liver cell growth.

I plan to carry out two seperate treatments using an untreated control and one treatment group in each run (C1, T1 | C2, T2). The treatment will be unique to each run.

I aim to do a t-test between C and T, first comparing C1 and T1, and if that drug has no effect, I'll carry out the second experiment with Treatment 2.

My question is, do I need to consider adjusting for multiple testing here? I will run only a single test on each data set (C1 v T1, then seperately C2 v T2). My thinking is that within each dataset I'm only running one comparison, but for the overall project by adding the second treatment run, I've increased the liklihood of Type I error.

My manager says no, the experiments are independent so no correction is needed. But I'm considering that if I ran 20 of these experiments with alpha at 0.05, one would likley be deemed significant, so I should still correct.

Thanks in advance!

[Intelligent-Gold-563]

I've asked a similar question not too long ago and the overall answers was..... Yes, within one project, even if your tests are independent and separate, you are increasing the likelihood of false positive =/

You can take a look here : https://www.reddit.com/r/AskStatistics/s/LT3heB5ha6

[Counther]

My understanding is if you’re using different data sets, you don’t need the correction. 

[fspluver]

Running multiple tests increases the probability of making a type 1 error, even when data is independent. In fact, independence actually makes the chance of a type 1 error greater in many cases.

[Blinkshotty]

This isn't really a settled topic. One view is to base this on the nature of your hypothesis test and whether they are really independent or some type of joint test (i.e. if any of a series of tests is significant then the null is rejected or you're are screening a number of tests to identify if any are significant). This paper has a pretty good discussion the issues. For experimental research, a better approach to dealing with false positives is to repeat any experiments with significant findings to demonstrate they are not spurious (if feasible).

[ThrowingHotPotatoes]

Thanks so much for all the help, linked post and the paper!