Fig. 1

An impossible dataset. The NIH Primer illustrates the idea that when data from males and females are pooled, sex differences can be masked. (A) depicts a graph featured in a slide from the Primer Module 3 (see Fig. S1A). On this slide, which is part of a quiz, learners are asked to draw a conclusion about whether the intervention had an effect. No statistical results are presented. The “correct” answer is that the intervention had no effect. In the explanation of the correct answer, learners are told that such a result would be “evidence for the null hypothesis.” (B) depicts a graph from the next quiz question (see Fig. S1B), which claims to contain “the data from the same experiment disaggregated by sex.” Learners are again asked to draw a conclusion without seeing the results of statistical tests. The “correct” answer is that the intervention had an effect. Our analysis of the dataset presented in (B) (see Supplemental Methods) shows that the sex difference in the control group in (B) would be one of the largest quantitative sex differences ever described in any species (Cohen’s d = 23.24). After the intervention in (B), the sex difference flips to what would again be one of the largest ever measured, but in the opposite direction. (C) We reconstructed the dataset shown in (B) (see Fig. 1 Supplemental Methods and Table S3) and plotted the data pooled by sex with accurate error bars, showing the impossibility that the dataset in (A) could be the same as in (B)