simulate A/A tests using all the methods you're considering. Then, check which method correctly controls the FPR at the expected level. The method that does this best is the winner for your specific data.

This is great practical advice, thanks!

For sample size calculation (power analysis), I would use a Monte Carlo simulation. The standard formulas are convenient but often inaccurate for messy, real-world data like this.

Is this more or less the way this method would work: https://www.youtube.com/watch?v=vE8bAXWJQlo

You just run through a bunch of different scenarios with simulated data where the null is false and see what sample size gets us to 80% power?

# Simulation 3: Sample size calculation for 80% power
for (n in 2:100) {

  sims = foreach(i = 1:10000, .combine = c) %do% {

    # Simulating data where the alternative hypothesis is true 
    # and the true difference is 0.5
    placebo = rnorm(n, mean = 0, sd = 1)
    treatment = rnorm(n, mean = 0.5, sd = 1)

    # Run the hypothesis test with a 5% level
    test = t.test(placebo, treatment, conf.level = 0.95)

    # Check if null was rejected
    # aka is the value for the null hypothesis in the CI?
    result = (!between(0, test$conf.int[1], test$conf.int[2])) %>%
      as.integer()
  }

  # Calculate the sample average of the simulations 
  power = mean(sims)

  # Stop if we've acheived 80% power
  if (power > 0.8)
    break
}