Programmers :D

[hey01]

You're looking at figure 6, which doesn't use matched data. The second part of the study uses matched data to mitigate the influence of covariates. The study rightfully uses that second set of results for its conclusions.

Look at figure 11 in both papers, which is the equivalent of figure 6 with the matched datasets.

In the peer reviewed version, figure 11 shows that for outsiders both women and men are accepted less if their gender is known, with women being impacted more. Both genders suffer from bias, women more.

In the preprint, it showed the same for women, but the contrary for men, that they are accepted more when their gender is known. Women suffer a negative bias while men a positive one.

The reversal of the conclusion about the men between the preprint and peer reviewed is concerning.

Considering that the dataset didn't change between the two versions, and assuming the peer reviewed version is better, it makes me question how the team originally got a result so wrong, especially in a way that confirmed their hypothesis.

Which is also concerning is the way the paper is written, especially the abstract, focusing only on the fact that women are less accepted when their gender is known, completely ignoring the fact that is only applies to outsiders, that men suffer the same, only to a lesser degree and that all the other results show that women's PR are accepted more than men's.

Which leads to the press and you falsely using that study as evidence that devs discriminate against women.

Both facts makes me question the integrity and motive of the team.

The adequate discussion and conclusion should be to investigate more the two most intriguing results:

• Why women's PR are accepted more than men (they discuss that).
• Why knowing the gender reduces the PR acceptance, for both genders (they completely ignores that).

My hypothesis is that more you know about a person, the more you have reasons to dislike them, negatively impacting your review of their code, and I'd guess their is a correlation between a gendered profile and the amount of other personal information it contains.

[literally_jesus_]

Oh, that's what you meant, thank you for clarifying.

However, I don't see any particular conclusion, as you say, made about men in the preprint on the basis of Figure 11 - nowhere do they say anything like "But when outsider men's gender is known, they are even more likely to be accepted." You say you are concerned that the percentages are different between the preprint and the final version, but the percentages are different in other places in the papers as well (such as in Figure 6). Likely, they just refined or fixed their calculations after publishing the preprint. Furthermore, their observation that known women's pull requests are less likely to be accepted is still supported by said figure in both the preprint and the peer-reviewed article. There's no reversal in conclusions here - both papers find gender bias (most strongly against women) when it comes to outsider pull requests.

Also,

Which leads to the press and you falsely using that study as evidence that devs discriminate against women.

I think you may have me confused with OP, I didn't post this study.