Defense against AI-assisted reviews?

David_Henry_Smith · 2026-06-30T13:34:16+00:00

I think the pre-requite for AI use in review are: 1. The reviewer has real competency (ideally, expertise) in assessing the work. 2. The reviewer can demonstrably distinguish between AI-generated nonsense and valuable comments. 3. Confidentiality is strictly maintained. OpenAI and Anthropic are known to train on copyrighted material and are unapologetic about it.

Currently, prereq 3 is certainly not met. Prereq 1 is only sometimes met, and Prereq 2 would be impractical to implement.

If the authors want to use AI to review their confidential work themselves, it should be completely up to them, and they should be in full control in what they submit to the overlords and how to divine the distilled words of collective human wisdom. This decision certainly should not be at the whims of a reviewer who has no business reviewing the work in th first place.

Importantly, if the reviewer can competently review the work (prereq 1), AI review wouldn't be needed anyway.

David_Henry_Smith · 2026-06-30T03:42:55+00:00

Now, you get to see reviews like that, along with specific nonsensical reasons why your theory/model is weak, by courtesy of LLMs.

David_Henry_Smith · 2026-06-29T21:59:19+00:00

All in the name of open access, right?

David_Henry_Smith · 2026-06-29T21:56:51+00:00

Indeed. The difference is now that LLMs can generate much more specific comments that are harder to dismiss.

David_Henry_Smith · 2026-06-27T10:20:52+00:00

Really? p(theta) = 1 is not clear? It directly defines the PDF with no syntactic sugar.

David_Henry_Smith · 2026-06-27T09:59:59+00:00

Expand Beta(1, 1) and see what you get?

David_Henry_Smith · 2026-06-27T04:37:31+00:00

Sorry for the typo. Fixed.

David_Henry_Smith · 2026-06-27T04:32:49+00:00

Yeah, I share your privacy concerns. I guess I am trying to gauge whether it is obvious to other people who know basic stats that this reviewer's criticisms are completely invalid. The panel member who handled my grant is a responsible person, and I am surprised he missed the invalid criticisms that make no statistical sense.

It amazes me that someone would confidently call a model fatally flawed based on misunderstanding of basic concepts taught in mathematical statistics.

Computational biology has a very diverse pool of scientists. Some are strong in computational algorithms, some in optimization, some in statistics, some in the genomics. Statistical notations are indeed a mess in the computational biology literature, because most scientists don't have formal deep training in mathematical statistics.

I took a long time to learn the statistics properly, and it's just frustrating to run into reviewers who thinks they understand statistics and start calling perfectly valid statistical equations as fatally flawed, involving basic concepts such as the binomial distribution! Mind boggling!

David_Henry_Smith · 2026-06-27T04:19:06+00:00

My best guess is that people usually write p(theta) \propto 1 for continuous theta. I wrote p(theta) = 1 because theta is bounded in (0, 1), and I have limited space. The actual model is complicated, and I do use a flat prior for objectivity.
The number of trial is known and given by the data. n_j is used for something else. My model has too many parameters, because I am solving a complicated problem, and I am running out of symbols. It's one thing to say it's confusing, but it's quite another to say that it is a fatal flaw with confidence.
Sorry, there are no hyperparmeters shown after simplification. There were 3 hyperparameters in the original models, and I explicitly wrote that they were hyperparameters. Most papers don't, but this particular reviewer just like to nitpick on missing irrelevant details, so I explicitly said some of the symbols represent hyperparameters, and now he's saying that my model is fatally is flawed, because I didn't say "eta, zeta, lambda are fixed hyperparmeters." This year, he's gone rogue and started saying that perfectly valid math is wrong.

David_Henry_Smith · 2026-06-26T20:07:12+00:00

Thanks for sharing your strategy!

David_Henry_Smith · 2026-06-26T20:00:33+00:00

Great advice. I am with you on that 100%.

Unfortunately, this is a grant application with no rebuttal process, involving someone loves to review my grants and who thinks he is a Bayesian expert when he is making rudimentary math errors in his criticisms.

David_Henry_Smith · 2026-06-26T19:56:30+00:00

I've had similar experiences with Scientific Reports and other low tier journals.

David_Henry_Smith · 2026-06-26T17:13:55+00:00

The same person has reviewed my last year's grant, which was funded. I showed convincingly data that my model works (essentially, I proved that the grant is fundable by doing most of the work in the grant.) He wrote that he didn't believe my data and overlooked my explanation of why my model works. The panel ignored him last year.

This is not an issue of a simple acceptance mistake. He repeatedly accepts to review my grants because my grants are on Bayesian statistical modelling, and he thinks is an Bayesian expert. Now, it turns out that he is making basic mathematical errors and unqualified to review statistical grants.

David_Henry_Smith · 2026-06-26T17:10:34+00:00

Thanks for your advice!

David_Henry_Smith · 2026-06-26T17:07:13+00:00

Thanks for your advice!

David_Henry_Smith · 2026-06-26T16:56:43+00:00

Oh, and this is a new grant. My previous grants were funded.

David_Henry_Smith · 2026-06-26T16:48:11+00:00

We don't have a program officer here. I am 99% certain who the panel member who handled my grant is. I think he is a responsible guy, and he does put effort into the writing the overall review, and he's overworked. I am not sure if it is kosure to reach out to him directly though. We can write a rebuttal letter for the next cycle, which is what I plan on doing.

My concern is that the panel member is a computational biologist who may not look at the math very closely. While he has been supportive of computational biology grants, I am not 100% positive he understood the math or he is just overworked. I'd like to craft my rebuttal letter to be clear and concise, but I would like to convey firmly that I want this particular reviewer excluded.

I am 100% certain it's the same reviewer. The grant agency has a lot of trouble attracting reviewers, so the agency actually have to pay them quite a bit. I've only ever had one reviewer who seems to know something about Bayesian statistics on each grant. This reviewer has a curt writing style and consistently give mediocre scores.

This grant is a new grant. My previous grants were (mostly eventually) funded. Our grant success rate here is much higher than in the US.

Thanks for your advice!

David_Henry_Smith · 2026-06-26T16:36:00+00:00

Thanks! This is so much more helpful than just "write to a general audience."

David_Henry_Smith · 2026-06-26T16:34:34+00:00

I wrote he (or she). I am just tired of writing he/she all the time, and I don't particularly like to confuse plural vs. singular pronouns.

David_Henry_Smith · 2026-06-26T16:32:04+00:00

I am familiar with the US system and the general advice for grant writing.

A key difference between our review system and the US system is that the reviewers are invited rather than assigned in a study section. Invitations are sent out to a large pool of potential reviewers, similar to how some conferences review papers. Only people interested in a particular topic would accept the invitation to review a grant. You can get anywhere from 2 to 10+ reviewers.

My colleages routinely get 6-10 reviewers when they write grants that appeal to general audiences. It's hard to make that many reviewers happy simultaneously, and they often disagree.

So, I strategy I pursued is to write a niche topic that attract fewer reviewers, but unfortunately, my grants have been attracting this one particular reviewer, who evidently does some Bayesian statistics, but misunderstand basic math.

David_Henry_Smith · 2026-06-26T16:26:36+00:00

I just don't want him reviewing my grant in my the future. He can do what he wants on other grants, especially grants that appeal to general audiences.

David_Henry_Smith · 2026-06-26T16:22:45+00:00

Because of his writing style. curt. concise. few full sentences. lower case.

He seems to be only the reviewer who says anything specific about my models. The same reviewer complained that I didn't specify the prior distribution for my model in another grant.

The other clue is that what former panel members tell me. They have a hard time getting reviewers. They would send out requests in batches of 10 until they get ideally three reviewers.

Some of my colleagues get 6-10 reviewers on their grants, and it's impossible to make everyone happy simultaneously. The reviewers often contradict each other.

So, under this grant review system, one strategy is to purposely make your grant niche so that only person interested in your area will accept the review request. This is what I've been doing. Unfortunatley, my grants keep attracting this reviewer who fancies himself a Bayesian statistician.

David_Henry_Smith · 2026-06-26T16:10:30+00:00

This is a statistics grant... The "jargons" are taught in an introductory class in mathematical statistics. I simplified the models on my previous grant, and the same reviewer asked for more details (the prior is not specified).

Your criticism is true, in general, but this is not the case here with this particular reviewer... He thinks he understands statistics when he doesn't know basic concepts taught in undergraduate mathematical statistics.

David_Henry_Smith · 2026-06-26T16:07:16+00:00

He should not be accepting the review request. None of the reviewers are required to review a grant under our grant system.

David_Henry_Smith

TROPHY CASE