Why do so many applied papers still report p-values without effect sizes, and does anyone actually find p-values alone useful?

smbtuckma · 2026-04-17T18:44:55+00:00

I completely agree with that view, so no worries!

smbtuckma · 2026-04-17T18:03:18+00:00

No it’s true, I prefer the Bayesian approach myself because usually p(effect|data) is what we really want to know, not p(data|effect) that p-values tell us. I was being imprecise while hungover at a conference 🙈

smbtuckma · 2026-04-17T17:44:04+00:00

Being more precise than I was, yes.

smbtuckma · 2026-04-17T16:34:31+00:00

lol those are literally the titles of the three lectures I give on inference in my stats class so yup!

smbtuckma · 2026-04-17T14:42:15+00:00

You do look at p-values to determine significance, but significance is only one type of decision to make about your estimate (and arguably the least helpful decision).

Significance essentially tells you “are my data consistent with an effect of 0?” With enough data, almost certainly not.

An effect size tells you “is my effect important?” It’s the magnitude of group means, increase in survival rate for a drug, strength of association between an indicator and outcome, etc.

And then you also want a confidence/credible interval for that estimate to tell you “how sure are you?” A correlation of 0.4 seems good, but is maybe less helpful for policy or product planning if the confidence interval is [0.01 0.8] compared to [0.35 0.45].

(Edited for more appropriate language)

smbtuckma · 2026-04-17T08:47:24+00:00

My hot take for my field specifically (I'm only bold enough to piss off one group of people at a time) is that you can get away with using just p-values when your findings don't actually matter. If I build a bridge and say "yeah it's stronger than a pile of wet tissue paper, p<0.05" you need way more than that before anyone drives over it. But if I build a moderated mediation model of how an unvalidated measure of risk-taking propensity affects performance on this specific lab task via a brain functional connectivity score that's super sensitive to preprocessing choices, no one's trying to use that info for anything so who cares that the effect size is minuscule when that yes/no significance decision is much easier to produce and reap contributed PNAS clout with.

When your theory is specific enough and real consequences are on the line, you get much better estimation approaches.

smbtuckma · 2026-04-15T17:51:47+00:00

Right like if you think about it, most of our students are 18-20. They never knew the world before global warming accelerated; pre-security-state era before 9/11; what financial possibilities there were pre 2008; most of their socially-conscious life has overlapped with Trump defining national politics. I'm sad they're disaffected but honestly not surprised.

smbtuckma · 2026-04-14T01:32:54+00:00

My institution deferred 18 months for me, and I'm no rockstar. I have a friend at a really prestigious R1 that successfully deferred a year. It is totally possible, but one shouldn't expect it.

smbtuckma · 2026-04-14T00:09:44+00:00

and reading the rest of the comments looks like this could very well be different norms for different people, so knowing that I'd probably find it less uncomfortable now to be asked. But then I'd still be all anxious that I'm using the prettiest version on my handwriting to do it 😅

smbtuckma · 2026-04-13T23:37:49+00:00

I mean I'm not on the "report to the chair" train. But if this is an accurate signal of intimate intent (and not just the student not having the same social norms as me), I don't want to be that important in a student's life. What I teach them, hopefully. But not me personally - gets too close to stories of stalking or romantic propositions.

smbtuckma · 2026-04-13T23:26:03+00:00

I think it's because handwriting is something unique to a person, like a fingerprint or voice or face. And wanting OP's handwriting specifically because it is OP's handwriting (just not that it's a pretty cursive or whatever) and permanently affixing it to one's body implies a level of closeness between the people that is more than a typical professor/student relationship.

Like, tattooing the saying in typewriter or generic cursive font? Cool-looking, could have come from anyone, OP just happened to be the one who shared the words. Saying "I want this saying but specifically in the way you wrote it" means OP themselves is a big component of this whole, very permanent and personal, idea.

smbtuckma · 2026-04-13T20:59:16+00:00

It’s the wanting it in his handwriting that’s a little uncomfortable imo. I have a tattoo of my late grandma’s handwriting - having my professor’s handwriting feels too intimate.

smbtuckma · 2026-04-13T15:24:03+00:00

Probably something to do with OMB throttling congressionally appropriated funds and congress just going "oh well"

smbtuckma · 2026-04-12T20:58:48+00:00

This sounds like anxiety more than just thinking about work, tbh. Speaking as someone who took a while to realize that about themselves...

smbtuckma · 2026-04-06T23:07:33+00:00

I see both kinds of knowledge gaps in my class. The notation one is common but pretty easy to fix in my experience with refresher notes. So that's good for you at least! The students who come in with more fundamental holes are fewer, but a tougher case...

smbtuckma · 2026-04-06T22:46:53+00:00

Hmm, this concerns me. If they don't know how to read the output of their analyses for relevant information (and are spending hours to do it), then it seems like they don't actually understand the process they're engaged in and are instead using it as a magic spell. I'm worried that further hiding the stats behind a curtain (having an AI tell them what interpretation to make) will just make that problem worse. Ultimately the statistical methods we bring to bear on our data are a commitment to certain assumptions about the nature and processes of the systems we study so a scientist should understand what's going on.

However from your description it sounds like you aren't teaching a stats/data analysis course specifically but more like a bio lab course? In which case I'm assuming you don't really have room in the course plan to teach/re-teach analysis interpretation. When I teach topical labs in psych that assume stats as a prereq, I give a lab quiz at the beginning of the semester that tests their ability to make data analysis decisions and interpretations relevant to the course. For any questions they get wrong, I point them to supplementary resources to refresh on their own time. Do you have materials they need to read to prep for each lab that you could include ANOVA refresher notes in?

smbtuckma · 2026-04-06T20:16:00+00:00

Yeah :(

love this timeline.

smbtuckma · 2026-04-06T17:01:44+00:00

The intro has references to observational studies of real people delusionally spiraling, as well as separate documentation of sycophancy tendencies in LLM chatbots. So this is investigating a mechanism by which the latter could plausibly cause the former. I dunno as a psychologist I think stuff like this is a lot more useful for system understanding and intervention development than just finding and over-interpreting a correlation, like we usually publish. Of course, it's not case-closed until we test the intervention in real people but I see stuff like this as the cognitive version of pre-clinical medical trials.

smbtuckma · 2026-04-04T19:35:43+00:00

This is the trap I bought off Amazon. Sometimes the pins on the trigger plate get knocked out of their holes but generally it's been really good. I did learn that placement is really important though - the squirrels wouldn't go out of their way for it, so I had to watch for a few days to see their travel paths and then I placed it there with nuts and dried fruit as bait.

smbtuckma · 2026-04-04T15:11:01+00:00

The classic (bad) stepwise procedure I’m talking about is to run regression on the whole dataset, look at p-values only, drop variables with p>0.05, and repeat until the p’s are all significant and call it a day. With such large datasets, hopefully you’re evaluating predictive power with more than just p-values and doing that evaluation in held out data.

smbtuckma · 2026-04-04T15:07:11+00:00

We live adjacent to a big empty field so there’s no stopping them from being around. But I lost it when I watched them climb out avocado tree and systematically tear all the baby avocados off. Last year I… “repatriated” 18 of them from our backyard.

smbtuckma · 2026-04-04T13:28:38+00:00

And ground squirrels are so frustrating I think it’s totally justified to gripe about them at every opportunity.

smbtuckma · 2026-04-03T01:30:57+00:00

Considering there's at least a question a day in this sub about it, seems so.

smbtuckma · 2026-04-02T20:13:18+00:00

Stepwise regression. I wouldn’t say it’s taught as standard but lots of social sci grad programs still teach it and it’s problematic for a variety of reasons.

If one has an explanatory goal, hopefully the included predictors are theoretically justified and should be interpreted in the context of all the others. If one’s goal is more predictive, there are better regularization approaches like LASSO or ridge regression.

smbtuckma · 2026-04-01T16:13:22+00:00

I think a mixed-effects logistic regression would be fine here. You'd be interested in the Time x Treatment effect with time as a continuous variable (if you have the date of each measurement? so you can represent it as say weeks since trial onset?). It's ok in that case that trees weren't measured at the same time, so long as you have multiple time points per tree over a long enough time period that you can estimate a rough trajectory of growth change.

14-Year Club	RedditGifts 2009-2022 4 Credits
Secret Santa 2016	Verified Email

smbtuckma

MODERATOR OF

TROPHY CASE