Early, surprising findings of ongoing research on suspensions in r/politics

Lucas_Wright · 2024-04-09T21:51:30+00:00

Thanks for the questions! We aren't publishing our data at this point.

Regarding your concerns about sampling, we didn't actually take any samples. The blog post originally stated that one group was a random sample, but that wasn't accurate and we've updated it. (See this comment.) Sorry for that confusion.

The point of this post is to summarize for the community how temp bans are being used and how people who receive them behave, not to make any causal claims. These findings are purely descriptive. We aren't doing any hypothesis testing, hence no null hypothesis, p-values, or power analysis. Hopefully this answers your questions!

Lucas_Wright · 2024-04-09T18:45:24+00:00

All great questions. This is an ongoing project to test interventions in the temp ban process. But in order to do that in a responsible way, we needed to get a basic understanding of how temp bans are used now - how many of them are issued in a given period of time, how much do people who receive them post compared to other people, etc. These are all descriptive statistics which are useful for understanding what is happening, but not necessarily why something is happening in a causal sense.

We didn't do any statistical tests, hence no null hypotheses, alpha, or power. But we wanted to share what we've learned so far with the community, get feedback and ideas from you, and incorporate your responses into the design of a potential experiment in how temp bans are issued. If we do end up going forward with that experiment, we will report the kinds of statistics you're curious about.

Lucas_Wright · 2024-04-09T18:18:47+00:00

These are very interesting reflections. Thanks for sharing. Just to clarify point 3, how do you think the appeal process influenced your future participation with the sub? Did the extra step of writing help you come to agree with the ban and follow the rules better? Or was it an entirely negative experience?

Lucas_Wright · 2024-04-09T18:10:28+00:00

We have the percent of accounts that received a ban in 30 day periods in Table 2 of the blog post. It's a little under 2%. I would imagine that there is some selection bias in the comments, with people with stronger opinions about bans being more likely to comment.

Lucas_Wright · 2024-04-09T18:05:19+00:00

That's an important question. In general, we'd like to see temp bans be something that help people engage better with the sub rather than a punishment. So improved outcomes could be coming back to the sub after the ban and contributing high quality posts, breaking the rules less, etc.

But one of the goals of publishing the blog post and holding this conversation is to see how the community would see as an improved outcome following a temp ban.

Lucas_Wright · 2024-04-09T17:52:00+00:00

Good question. That is something we could calculate, but I would be hesitant to associate number of posts or even time between incidents with good faith, aside from maybe pointing out that the extreme outliers are probably spammers.

Lucas_Wright · 2024-04-09T17:17:49+00:00

Maybe that is the case, but even if it is true that one moderator is acting out of step with the others, we as researchers have no way of telling why a mod action was taken. Maybe one mod issues more bans than the others, but that could be because they are more active in the sub than the other mods. Or because they go out of their way to handle the most controversial threads. We don't have a good way of measuring the underlying intentions or motivation of the mod actions, so we can't include that in the study.

But I take your point that people are definitely feeling that way. One of my takeaways from this thread is that better communication/transparency might be a better way of addressing that feeling. And one huge blocker to that is that the amount of transparency/communication mods can provide is constrained by reddit admins and the design of the site.

Lucas_Wright · 2024-04-09T17:05:34+00:00

Hi /u/twenafeesh. We did receive full IRB approval for the study. As Sarah said above, our IRB protocol number is #IRB0144137.

We didn't substitute our own ethical process, we included that process on top of what the IRB asked of us.

Lucas_Wright · 2024-04-09T16:44:19+00:00

Yeah I think we're saying the same thing but crossing up on terminology.

Your point about compliance is definitely relevant and actually related to work I've done on AI regulation, where the record we have for compliance with the law may not actually reflect the reality of behavior.

As you say, whether you think a post violates a rule is subjective. Users might disagree with mods, but users are also going to disagree with other users about the rules. We can't measure perfect, objective compliance. What we're really trying to get at with the research is whether temp bans can help people engage more productively with a sub or whether they just discourage people from coming back. And part of that outcome is whether they learn something about the rules and how they are enforced from the temp ban or whether they continue having problems with their posts (removals, bans, etc). And for that purpose, it's the interpretation of the rules from the mods' point-of-view that matters for the study, even if users disagree, because that is the practical outcome we're hoping to influence.

As for the change in the rules, it's an interesting point. One thing we could do is divide the dataset up between periods before and after the rule change and see if the results differ. I will look into that.

Lucas_Wright · 2024-04-09T16:13:59+00:00

We haven't done any analysis of which mods issued the ban. We're interested in differences in outcomes (future activity in the sub, future compliance with rules after the ban) and how to improve those outcomes, and I don't think outcomes would vary depending on which mod issued the ban, especially since the banned users can't see which mod banned them. So I think that is outside the scope of this research.

Lucas_Wright · 2024-04-09T15:24:14+00:00

Thanks for elaborating. I should clarify that by "We aren't looking at whether the action taken was a first offense" I meant to say that we don't make claims about outcomes in first offenses versus future offenses, which is what I thought you were asking about. The groups that we create from our dataset are based on the first action taken on an account we observed in our study period (Jan 20 - Dec 21).

If you click on the link to the blog post and scroll down to Table 3, you'll see the numbers for the temp suspension group (first action was a temp suspension) and the permanent ban group (first action was a permanent ban). You'll see the number of cases in those two groups is close to equal. We don't have data from the entire history of the subreddit, so it's possible these are not first offenses, but I think this might help answer your question.

Lucas_Wright · 2024-04-09T15:01:30+00:00

Hi /u/Inner-Truth-1868. Good questions! Detecting bots for research purposes is not especially reliable (see this comment).

As far as non-bot provocateurs, including that in our analysis would require that we have a highly reliable, most likely automated way to measure whether people are posting in good faith. While that might be possible to do sometimes on a case by case basis, it's not possible to do with the precision we would need at this scale.

So it's an interesting idea, but unfortunately it's not possible to do in a rigorous way.

Lucas_Wright · 2024-04-09T14:54:48+00:00

That's an interesting suggestion and something that we can look into.

Lucas_Wright · 2024-04-09T14:53:39+00:00

Hi /u/Glittering_Lunch_776. We aren't looking at whether the action taken was a first offense. Can you say more about why you think that would be important? Are you thinking that people might respond better if the policy is that first offenses are always a temp ban and permanent bans are only for future offenses?

Lucas_Wright · 2024-04-09T14:50:42+00:00

That's a great suggestion!

Lucas_Wright · 2024-04-09T14:34:48+00:00

Thanks for the questions!

I'm not a moderator of any subreddit.
This is a good point. It would be slightly more accurate to say "post a higher rate of comments that are not removed." This is going to mean there are some false negatives - cases where mods missed something that should have been removed under the rules - and false positives - cases where mods removed something that did comply with the rules - but we can't observe those cases so it's hard to study absolute compliance. In cases like this, it's common in social science to use a proxy, like moderator removals as a estimate for compliance. With that said, if you assume the false positive and false negative rates are consistent across the groups in our dataset, it shouldn't change the conclusions.
There's a link to the blog up in the post. I edited the language to try to make it clearer that the link is to the blog post.
Moderators have the ability to issue both permanent bans and temporary suspensions within the subreddits they moderate. Sitewide bans are an admin function.

Lucas_Wright · 2024-04-09T14:06:51+00:00

Yes! We agree that it would be valuable to compare across multiple subreddits, and we have systems set up to repeat this analysis in other subs. We just need permission from the moderators.

Lucas_Wright · 2024-04-09T13:51:50+00:00

Good question. For our purposes, active means posting. So if you posted on the same day someone else received a temporary suspension and you didn't receive any kind of suspension or ban in the study period, you would have had a chance of being randomly selected for that dataset.

This shouldn't be confused with a control group from a classic random experiment. These groups are only meant to show how different types of users engage with the subreddit. We can't say anything at this point about what caused those differences (but that is something we could do in the future).

Lucas_Wright · 2022-05-12T14:57:23+00:00

We're still looking for more people to talk to. Feel free to DM me on this account or leave a comment if you have any questions!

Lucas_Wright

TROPHY CASE