- Open Access
Effectiveness of online learning on health researcher capacity to appropriately integrate sex, gender, or both in grant proposals
© The Author(s). 2018
- Received: 12 July 2018
- Accepted: 15 August 2018
- Published: 29 August 2018
To describe the effectiveness of online learning to augment academic capacity to consider sex and gender in the conduct of basic science, clinical research, and population health studies.
The analysis compares pre- and post-test scores from 1441 individuals who completed the Canadian Institutes of Health Research Institute of Gender and Health’s interactive e-learning modules between February 2016 and May 2017. The tests measured knowledge, self-efficacy, and self-reported intent to change behavior for three competencies: (1) the ability to appropriately define and distinguish between sex-related versus gender-related variables, (2) the application of methods for integrating sex and gender, and (3) the critical appraisal of sex and gender integration in the design, methods, and analysis plan of research proposals and publications.
Of the 543 individuals who completed the basic science module, 62% demonstrated improved knowledge, and 86% increased self-efficacy across all competencies. Gains in knowledge and self-efficacy also occurred among 84% and 77% of completers of the human data collection module (n = 463) and among 73% and 82% of those who completed the secondary data analysis module (n = 435). In aggregate, 95% of participants reported an intent to change their behavior with respect to sex and gender in health research.
Interactive online learning combined with feedback and self-assessment results in improved knowledge and self-efficacy for integrating sex and gender in health research.
New funding agency policies and journal instructions for authors require integration of sex as a biological variable and gender-related social factors in health research [1–4]. The highest standards of excellence in science—including rigor, reproducibility, transparency, and inclusion—are driving this change in order to ensure that all patients benefit from research investments in an equitable fashion. Cell biologists are being asked to record the sex of their cells . Animal experiments are expected to factor sex as a biological variable in the design and methods of the study . Clinical trialists must document the sex and gender of study participants and disaggregate the data in their analyses . Health systems research calls for gender—the socially constructed roles, behaviors, activities, and attributes that a given society considers appropriate for males, females, and other genders—to be a core component of interventions, programs, and policies . Even implementation research acknowledges the advantage of addressing sex and gender to increase the uptake of healthcare solutions . Ultimately, the vision is for evidence-informed sex- and gender-specific recommendations to be integrated into research and clinical practice in a way that best tailors care to the individual patient according to sex, gender, and other identity-related characteristics [9, 10].
The most effective way to educate academics on how to appropriately consider sex and gender as a standard practice in health research remains unknown. Data indicate that uptake has been slow and irregular [11–13]. Barriers include problems with inconsistent terminology, difficulties in applying the concepts of sex and gender, failure to recognize the impact of sex and gender, and uncertainty about how to operationalize these concepts in data collection, analysis, and practice . Attempts to improve research integrity in other areas of science, such as preventing misconduct and reducing publication bias, suggest that overcoming inertia and changing cultural norms are major obstacles [14–16].
Literature on behavior modification among academics points to the success of multimodal education and training combined with interactive audit and feedback [17–19]. The benefits of e-learning are well documented in terms of increased accessibility to education, efficacy, cost-effectiveness, learner flexibility, and interactivity . The assessment of sex and gender in research protocols has never been tested through e-learning and holds promise as a vector for standardizing and building capacity among health researchers across geographic location and career stage.
In 2015, the Institute of Gender and Health of the Canadian Institutes of Health Research developed a series of Internet-based interactive e-learning modules to improve knowledge, skills, and attitude gaps related to sex and gender considerations in health research (http://www.cihr-irsc-igh-isfh.ca). These free, 45-min pedagogical courses were intentionally designed to help researchers and peer reviewers recognize sex- and gender-related mechanisms in health, identify methods for integrating sex and gender variables in health research settings, and critically appraise research protocols and publications based on the integration or omission of sex and gender. Three separate courses were developed: (1) for biomedical research involving animal, cells, or tissues; (2) for primary data collection in research involving human participants; and (3) for the analysis of data from human participants. This report describes the effectiveness of the three sex and gender e-learning modules in supporting improved knowledge, self-efficacy, and behavioral intent among trainees, researchers, peer reviewers, and government employees who completed the courses.
A quasi-experimental study design was used, comparing pre- and post-questionnaire data from the same participants, prior to and after completion of the online courses.
Participants were primarily health researchers, peer reviewers, trainees, research support staff, and government employees. Participants from Canada, the USA, Europe, and Asia completed the modules, either in French or in English. The data presented in this report is from individuals who completed at least one of the three online courses and submitted full data for the pre- and post-evaluations in English or French.
Recruitment of participants occurred through three different channels. The first was through email, newsletters, and social media to researchers already interested in sex and gender science, via a subscription to the Institute of Gender and Health’s communication list. The second method of recruitment was to other researchers through different organizations’ email lists, through presentations and flyers at national and international health research conferences, and via presentations to the chairs of all peer review committees for the Canadian Institutes of Health Research. The third method of enrolling participants included a mandatory requirement for grant applicants to complete one of the training modules in order to be eligible for specific funding opportunities supported by the Institute of Gender and Health. Examples of funding opportunities were catalyst grants to stimulate inclusion of sex as a biological variable for researchers who were previously not accounting for sex or team grants for personalized health research. To support the appropriate integration of sex and gender into their proposals, nominated principal applicants were required to submit a certificate of completion for one of the learning modules to be eligible for funding. Participants were not asked to indicate why they decided to complete the training module, so each individual’s motivation for engaging with the e-learning modules remains unknown.
Completion of one of the online learning modules was required in order to access the post-test. Only data from users who fully completed one of the modules and only responses from their pre-test and post-test questions are included in this analysis.
Development of the e-learning modules
The content and competencies targeted by each module were informed by consultation with members of the Advisory Board of the Institute of Gender and Health as well as external stakeholders and experts.
Each module was designed to promote three key competencies. The first was the ability to define and distinguish between sex-related and gender-related variables in a health research context. The second was the capacity to recognize when and why sex and gender are relevant considerations in health research and to identify and apply methods for integrating sex and gender in research. The third was to critically appraise research grant proposals and publications on the basis of the integration (or omission) of sex and gender considerations.
Measurement of knowledge, self-efficacy, and behavior change intent
The pre- and post-test assessments each consist of six multiple-choice questions testing knowledge and skills, and three self-efficacy questions. For each module, the three competencies are evaluated with two knowledge questions and one self-efficacy question each. The knowledge questions differ based on the content of the module. The self-efficacy questions are identical, modeled on Bandura’s social-learning theory, and measure the confidence a person has in their ability to perform specific tasks or behaviors related to each of the three competencies (ability to distinguish sex- and gender-related variables, ability to apply sex and gender methods in research, and the ability to critically appraise the appropriate integration or omission of sex and gender in protocols and publications) . Phrasing of the self-efficacy items and the response scaling was done in accordance with Bandura’s Conceptual Model of Self-Efficacy and his Guide for Constructing Self-Efficacy Scales . All self-efficacy items begin with “How confident are you that you can …” followed by a description of the specific competency. Response options are presented as per Bandura’s guide on a 10-point horizontal visual analog scale with two anchors: 0 indicating “not at all confident I can do” and 10 indicating “extremely confident I can do.”
Both the pre- and post-test assessment questions are identical for each module. Only partial feedback is provided after completing the pre-test (i.e., the number of correctly answered knowledge questions). The correct responses to the knowledge questions are shared only after completing the post-test. As users are asked to self-rate self-efficacy for each competency area, three distinct pre- and corresponding post-test self-efficacy scores are available for each completed module.
Intent to change behavior was queried with a single item after completion of the post-test for each module. Participants were asked to endorse one of two response options to the statement “With respect to the approach I use in my own research program and publications for integrating sex and gender, I intend to: 1) Change the way I account for sex and gender; or 2) Not make any changes with respect to the way I integrate sex and gender.” Perceived added value of the training modules was also assessed with a single item after completion of the post-test. Participants were asked to endorse one of two response options to the statement “With respect to improving my knowledge and skills for integrating sex and/or gender in research this training module: 1) Taught me new knowledge and skills; or 2) Did not teach me anything new.”
Demographic data (age, occupation, country of origin) from each registered user were obtained prior to completion of the pre-test for each module.
To assess knowledge gain, the number of correct responses on the knowledge pre- and post-test questions was calculated with 95% confidence intervals, for each competency, for each of the three modules. Comparisons between pre- and post-module test scores were performed using paired t tests. To assess gains in self-efficacy, self-rated scores were compared prior to and after completion of the module, with paired t tests. The distribution of participants who increased knowledge and self-efficacy, those who had a decrease in score and those that remained unchanged, were determined, along with 95% confidence intervals. Logistic regression was used to assess the odds of a change in behavioral intent due to increased knowledge or self-efficacy test scores, with adjustment for baseline scores. Linear regressions were used to assess whether participant demographics affected a change in score. All data were analyzed using R software version 3.4.0 (R Core Team (2017), https://www.R-project.org/).
Descriptive characteristics of training module participants
n = 543 (%)
Data collection in humans
n = 463 (%)
Analysis of human data
n = 435 (%)
Researcher and peer reviewer
Asia or Europe
Change in knowledge and self-efficacy
Distribution of changes in training module test scores
% (95% confidence intervals)
Data collection in humans
% (95% confidence intervals)
Analysis of human data
% (95% confidence intervals)
61.7% (57.5%, 65.7%)
84.1% (80.4%, 87.1%)
73.1% (68.7%, 77.1%)
3.7% (2.4%, 5.7%)
4.1% (2.7%, 6.4%)
5.6% (3.8%, 8.2%)
34.6% (30.7%, 38.7%)
11.8% (9.1%, 15.1%)
21.3% (17.7%, 25.4%)
85.8% (82.5%, 88.6%)
76.5% (72.3%, 80.2%)
81.6% (77.5%, 85.1%)
2.5% (1.5%, 4.3%)
7.6% (5.5%, 10.5%)
7.2% (5.1%, 10.2%)
11.7% (9.2%, 14.7%)
15.9% (12.8%, 19.6%)
11.2% (8.5%, 14.7%)
Behavioral intent and perceived added value
Association between gains in knowledge and self-efficacy on self-reported intent to change the way sex and/or gender is accounted for in research
OR, (95% CI)
Data collection in humans
Analysis of human data
Increase in knowledge scoreb
Increase in self-efficacy scorec
Effect of participant characteristics on improvement in test scores
Neither participant age, occupation, nor country of origin affected a change in post-test knowledge or self-efficacy scores, indicating that improvements were robust across learner profiles (data not shown).
E-learning was effective at increasing knowledge and self-efficacy for integrating sex and gender in health research and reporting. The majority of participants regardless of age or discipline acquired the three targeted competencies—(1) the ability to appropriately define and distinguish between sex-related versus gender-related variables, (2) the application of methods for integrating sex and gender, and (3) the critical appraisal of sex and gender integration in the design, methods, and analysis plan of research proposals and publications.
We believe there is a compelling rationale to mainstream the integration of sex and gender in health research and education. Not only are health research funding agencies, such as the National Institutes of Health and the Canadian Institutes of Health Research, calling for research proposals to account for sex and/or gender, but students expect it and patients deserve it [1, 2, 9, 23]. Sex-specific medication dosing is already recommended for drugs such as zolpidem and desmopressin [24, 25], and many physiologic parameters differ by sex . As the body of evidence grows in favor of sex- and gender-specific pathophysiologic mechanisms, the management of patients for a variety of health conditions will need to keep pace with the changing nature of science .
Several resources are available to help educators embed concepts of sex and gender into graduate and postgraduate curricula [27, 28]. However, to our knowledge, the Institute of Gender and Health e-learning modules are the first to undergo rigorous testing for evaluating changes in knowledge and self-efficacy. E-learning is flexible, with content that is dynamic, changing, self-paced, and that ultimately enhances the sustainability of knowledge transfer . The approach is ideal for standardizing sex and gender integration across settings and countries. Furthermore, the modules are free and widely accessible. To date, over 3000 academics, students, and public servants have completed the e-learning modules. Although e-learning cannot replace in-person mentoring and coaching, educators and supervisors can use the modules to support curriculum development efforts across health research disciplines.
A strength of this study was objective assessment of changes in knowledge and self-efficacy as a result of completing the e-learning modules. Gains in knowledge were statistically significant and provided clear evidence of participants having acquired competencies around defining and distinguishing sex and gender, applying methods to account for sex and gender in research, and appraising research protocols and publications. Changes in self-efficacy are harder to interpret, as they represent an individual’s judgment about their level of competence or ability to successfully perform a behavior . As a primary construct of social cognitive theory, self-efficacy is one of the constructs most frequently used to predict, explain, and change behaviors . Although self-efficacy scores improved in a statistically significant fashion in this evaluation, they rarely reached a score of 10 (full confidence). Interestingly, improvements in self-efficacy, but not knowledge, were associated with self-reported behavioral intent to change the way sex and gender was accounted for in research.
There are limitations to the current analysis. First, although the format of self-efficacy assessments is well accepted and valid, the knowledge questions in the pre- and post-tests were not previously tested and may not capture adequately the sex and gender competency constructs under study. Second, we queried self-reported behavioral intent as a single item question at the end of the post-test for each module. Behavioral intent is a subjective measure and is likely correlated with applicants’ motivation to complete the modules and possible propensity for e-learning. We do not know if completion of the modules translated into actual, objective changes in practice or future impact in a given field. The methods for recruiting participants may have led to enrolment of a biased sample of researchers already interested in the field of sex and gender in science, which may explain the relatively high pre-test scores for some of the knowledge items as well as the high rate of endorsement of the statement about changing behavior. Finally, assessments occurred once, immediately after completion of the modules. We did not assess whether the observed gains in knowledge and self-efficacy were sustainable over time. However, as most students’ satisfaction questionnaires are based on self-report, we believe that the methods used to assess the e-learning modules compare favorably with other course evaluations, with over 94% of completers endorsing the modules as being helpful for increasing their skills and understanding.
Funders, academic institutions, and professional societies are uniquely positioned to leverage continuing education to enhance the quality of health research and practice. As such, the Institute of Gender and Health’s e-learning modules have become a mandatory eligibility requirement for applicants submitting research proposals to an increasing number of priority-driven funding competitions in Canada. Promoting evidence-based capacity building methods that have been shown to increase knowledge and self-efficacy for integrating sex and gender in academic activities is an active step towards enhancing rigor, reproducibility, and inclusion. It is our hope that the e-learning modules will contribute to improving the uptake of sex and gender considerations in health research, policy, and practice.
We wish to thank Meridith Sones for assistance with the design and content of the biomedical module. Peter Smith wrote the outline and contributed examples to the module on secondary data analysis in humans. Patrice Voss aided with background material. Lingrui Meng conducted the data analyses. Gary Wang designed Fig. 1.
Availability of data and materials
The data is available from the authors upon request.
CT designed the study, interpreted the findings, wrote the manuscript, and approved the final version. KvH aided with the data collection and analysis, reviewed the manuscript, and approved the final version for submission.
Ethics approval and consent to participate
The study uses anonymous data from individuals who completed the online training modules. Approval was sought and obtained from the Canadian Institutes of Health Research for publication. The views expressed in this paper are those of the authors and do not necessarily reflect those of the Canadian Institutes of Health Research or the Government of Canada.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Clayton JA, Collins FS. Policy: NIH to balance sex in cell and animal studies. Nature. 2014;509(7500):282–3.View ArticlePubMedPubMed CentralGoogle Scholar
- Duchesne A, Tannenbaum C, Einstein G. Funding agency mechanisms to increase sex and gender analysis. Lancet. 2017;389(10070):699.View ArticlePubMedGoogle Scholar
- Clayton JA, Tannenbaum C. Reporting sex, gender, or both in clinical research? JAMA. 2016;316(18):1863–4.View ArticlePubMedGoogle Scholar
- Heidari S, Babor TF, De Castro P, Tort S, Curno M. Sex and gender equity in research: rationale for the SAGER guidelines and recommended use. Res Integr Peer Rev. 2016;1(1):1–9.View ArticleGoogle Scholar
- Shah K, McCormack CE, Bradbury NA. Do you know the sex of your cells? Am J Physiol Cell Physiol. 2014;306(1):C3–18.View ArticlePubMedGoogle Scholar
- Clayton JA. Studying both sexes: a guiding principle for biomedicine. FASEB J. 2016;30(2):519–24.View ArticlePubMedGoogle Scholar
- Morgan R, George A, Ssali S, Hawkins K, Molyneux S, Theobald S. How to do (or not to do)… gender analysis in health systems research. Health Policy Plan. 2016;31(8):1069–78.View ArticlePubMedGoogle Scholar
- Tannenbaum C, Greaves L, Graham ID. Why sex and gender matter in implementation research. BMC Med Res Methodol. 2016;16(1):145.View ArticlePubMedPubMed CentralGoogle Scholar
- Tannenbaum C, Clow B, Haworth-Brockman M, Voss P. Sex and gender considerations in Canadian clinical practice guidelines: a systematic review. CMAJ Open. 2017;5(1):E66–73.View ArticlePubMedPubMed CentralGoogle Scholar
- Johnson JL, Greaves L, Repta R. Better science with sex and gender: facilitating the use of a sex and gender-based analysis in health research. Int J Equity Health. 2009;8:14.View ArticlePubMedPubMed CentralGoogle Scholar
- Johnson J, Sharman Z, Vissandjée B, Stewart DE. Does a change in health research funding policy related to the integration of sex and gender have an impact? PLoS One. 2014;9(6):e99900.View ArticlePubMedPubMed CentralGoogle Scholar
- Philips SP, Hamberg K. Doubly blind: a systematic review of gender in randomised controlled trials. Glob Health Action. 2016;9:29597.View ArticleGoogle Scholar
- Day S, Mason R, Lagosky S, Rochon PA. Integrating and evaluating sex and gender in health research. Health Res Policy Syst. 2016;14(1):75.View ArticlePubMedPubMed CentralGoogle Scholar
- Marusic A, Wager E, Utrobicic A, Rothstein HR, Sambunjak D. Interventions to prevent misconduct and promote integrity in research and publication. Cochrane Database Syst Rev. 2016;4:MR000038.PubMedGoogle Scholar
- Carroll HA, Toumpakari Z, Johnson L, Betts JA. The perceived feasibility of methods to reduce publication bias. PLoS One. 2017;12(10):e0186472.View ArticlePubMedPubMed CentralGoogle Scholar
- Wager E, Williams P, Project Overcome failure to Publish nEgative fiNdings Consortium. “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ. 2013;347:f5248.View ArticlePubMedPubMed CentralGoogle Scholar
- Chauhan BF, Jeyaraman MM, Mann AS, et al. Behavior change interventions and policies influencing primary healthcare professionals’ practice-an overview of reviews. Implement Sci. 2017;12(1):3.View ArticlePubMedPubMed CentralGoogle Scholar
- Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259.Google Scholar
- Taylor D, Hamdy H. Adult learning theories: implications for learning and teaching in medical education: AMEE Guide No. 83. Med Teach. 2013;35(11):e1561–72.View ArticlePubMedGoogle Scholar
- Sinclair P, Kable A, Levett-Jones T. The effectiveness of internet-based e-learning on clinician behavior and patient outcomes: a systematic review protocol. JBI Database System Rev Implement Rep. 2015;13(1):52–64.View ArticlePubMedGoogle Scholar
- Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84(2):191–215.View ArticlePubMedGoogle Scholar
- Bandura A. Guide for constructing self-efficacy scales (revised) 2006. Available at https://www.researchgate.net/publication/233894825_Guide_for_Constructing_Self-Efficacy_Scales_Revised. Accessed 16 June 2018.Google Scholar
- Tannenbaum C, Moineau G. Innovative levers for sustainable integration of gender medicine into medical school curricula. Biol Sex Differ. 2016;7(Suppl 1):41. eCollection 2016View ArticlePubMedPubMed CentralGoogle Scholar
- Greenblatt DJ, Harmatz JS, Singh NN, et al. Gender differences in pharmacokinetics and pharmacodynamics of zolpidem following sublingual administration. J Clin Pharmacol. 2014;54(3):282–90.View ArticlePubMedGoogle Scholar
- Liu J, Sharma N, Zheng W, et al. Sex differences in vasopressin V2 receptor expression and vasopressin-induced antidiuresis. Am J Physiol Renal Physiol. 2011;300(2):F433–40.View ArticlePubMedGoogle Scholar
- McGregor AJ, Beauchamp GA, Wira RC, Perman SM, Safdar B. Sex as a biological variable in emergency medicine research and clinical practice: a brief narrative review. West J Emerg Med. 2017;18(6):1079–90.View ArticlePubMedPubMed CentralGoogle Scholar
- Miller VM. Why are sex and gender important to basic physiology and translational and individualized medicine? Am J Physiol Heart Circ Physiol. 2014;306(6):H781–8.View ArticlePubMedPubMed CentralGoogle Scholar
- Plank-Bazinet JL, Sampson A, Miller LR, et al. The science of sex and gender in human health: online courses to create a foundation for sex and gender accountability in biomedical research and treatment. Biol Sex Differ. 2016;7(Suppl 1):47. eCollection 2016View ArticlePubMedPubMed CentralGoogle Scholar
- Andrews R. Does e-learning require a new theory of learning? Some initial thoughts. J Educ Res Online (JERO). 2011;3:104–21.Google Scholar
- Glanz K, Bishop DB. The role of behavioral science theory in development and implementation of public health interventions. Ann Rev Public Health. 2010;31:399–418.View ArticleGoogle Scholar