A total of 23 of 24 individuals who accepted an initial invitation to attend the workshop responded to the survey. Of the 23 respondents, 3 were men and 20 were women. The estimated ages of respondents ranged from the late-20s to late-60s. Respondents came from a variety of academic, clinical, health policy and other experiential backgrounds and included university-affiliated and independent researchers, research coordinators and health-related professionals, some of whom were affiliated with non-governmental health organizations. Systematic reviews were used by respondents as sources of evidence for academic work, clinical practice, and/or policy development. Of the 23 respondents, 14 had experience with systematic reviews either as members of teams who conduct reviews or as contributors or advisors concerning review content or methods. Most were members of Cochrane and Campbell Collaborations’ methods and review groups and several were affiliated with other research organizations conducting reviews. Nine of the 23 respondents indicated a primary expertise in the area of sex/gender analysis. The respondents’ combined experience with systematic reviews and/or sex/gender was extensive and they formed a multi-disciplinary group with diverse academic backgrounds, approaches, and discourses.
Respondents provided rich and detailed answers to the questions, placing particular emphasis on the identification of challenges and thorny issues. No respondents offered examples of how they addressed these issues in their work. All respondents identified expected outcomes from the workshop. A summary of the results is provided in the following sections.
Challenges and thorny issues
Analysis of the data revealed a number of common themes among the perceived challenges to integrating considerations of sex/gender in systematic reviews. These have been organized into four main interrelated categories: 1) conceptual challenges; 2) methodological challenges; 3) challenges related to availability and quality of data; and 4) practical and policy challenges.
Conceptual challenges
Clarifying the concepts of ‘sex and gender’ and putting concepts into practice in systematic reviews were identified as significant challenges. Respondents noted the tendency for sex to be used as a proxy for gender and that the terms sex and gender were used interchangeably:
“One tricky thing has been the lack of clarity around whether it is gender or sex or both being reported in primary studies and in reviews. This makes us have to use the ever-ambiguous ‘and/or’ much of the time” (Respondent #6).
“Some reviewers (and other clinical researchers) may still be unclear about the concepts of ‘sex’, ‘gender’, ‘equity’, ‘race/ethnicity’, etc. and how to incorporate these” (Respondent #21).
Others pointed to the complexity of the concepts of sex and gender, indicating that they are not single variables, but interrelated biological and social processes that in many ways have yet to be understood. Respondents stressed the need for communication between sex/gender experts and systematic reviewers to provide clarity and guidance on capturing the nuanced and intersecting nature of sex/gender particularly within the context of systematic reviews.
Beyond the definitional challenges, respondents called attention to broader conceptual challenges that often manifest within interdisciplinary projects. Specifically, respondents raised concerns about the construction of knowledge and ‘what counts’ as evidence and the challenges these present to incorporating sex/gender analysis into systematic reviews. The absence of sex/gender considerations within systematic reviews was traced back to what influences how knowledge is produced:
“The biggest challenges are much more fundamental and have to do with the way that we arrive at decisions as to what is important for us to study, why it is important for us to study, and how we determine the way to study and ultimately produce evidence” (Respondent #3).
“The emergence of an evidence-based medicine approach has altered…our understanding of how to identify credible evidence and how to synthesize that evidence to inform medical practices as well as policies and programs… This approach to evidence, knowledge and policy-making has profound effects on who is authorized to provide credible accounts, what counts as credible evidence, and what counts as credible synthesis and translation” (Respondent #16).
The assumptions about ‘credible’ evidence were seen to have particular implications for the inclusion of sex/gender analysis in systematic reviews because sex-disaggregated data and sex/gender differences and similarities are often not taken into account in the research design, conduct, data analysis, and reporting of randomized controlled trials that form the basis of most reviews. In studies that include both men and women, conclusions are often presented as applicable to all ‘subjects’. This is, in part, related to a history of excluding women in adequate numbers from many clinical trials and the assumption that findings from studies of men could be extrapolated to women [48]. As respondents noted:
“Over time, I realised that women were being excluded from the development of evidence in a variety of ways… (and this had implications for) who were scientists, who defined the questions, who were subjects in research, what conditions were studied…” (Respondent #11).
“[My] concern is a lack of examination of sex/gender and other diversity within systematic reviews. There are important differences but does the current methodology hide those differences or fail to examine them… (Respondent #14).
Dealing with diversity and heterogeneity of populations was a significant methodological challenge described by respondents as detailed below.
Methodological challenges
Methodological issues generated the largest amount of commentary. Some respondents expressed concerns about a lack of transparency in clinical trial design, outcomes, and reporting current practices which have implications for the quality of systematic reviews:
“There is a lack of transparency about clinical trial design, including a priori outcomes and analyses, or poor quality protocols. There is insufficient reporting of methodologic information in current trial registries…” (Respondent #21).
The quality of reporting in primary studies has a cascading effect on reviews and on the analyses that can be completed:
“There are many perplexing issues with regards to conduct of systematic reviews. The most profound is quality of reporting. The quality of reporting directly leads to the comprised validity of systematic reviews, as well as contributing to the assessment of clinical heterogeneity which is a complex issue often overlooked by reviewers.” (Respondent #10).
In particular, respondents noted that sex and gender and other health determinants are often neglected in the single studies that form the basis of reviews. They identified major challenges to both primary studies and systematic reviews:
“…determining ways of measuring equity and bias, challenges of being able to capture complex social processes in what we mean by incorporating sex/gender.” (Respondent #5).
“… [the] question to ask is both related to quality of the research being reported and sorting out when, if, or how sex and gender matter.” (Respondent #11).
“There needs to be more work undertaken to explore how to identify and synthesize evidence on interventions to address structural determinants of health/health equity.” (Respondent #16).
An approach suggested to address these challenges is through use of a range of data sources and a variety of research designs:
“I’m wondering about the methodology and quality of the reviews that incorporate these different research designs and how we can use all the literature, not just the quantitative…to ensure a comprehensive overview of the literature, including grey literature, when we are making context-sensitive, evidence-based decisions.” (Respondent #14).
Respondents noted a lack of available tools, checklists, and/or outlines for conducting sex/gender analysis within primary studies and systematic reviews. Tools and checklists are practical aids that incorporate recognition and understanding of what sex/gender can mean when applied into practice, and were seen as essential for those new to the concepts of sex/gender. It was recognized, that “tools are still in development and lagging behind the theoretical arguments for inclusion of these issues” (Respondent #9). While tools are essential, caution was expressed about relying too heavily on a tool to conduct complex analyses:
“I struggle with the fact that I have not been able – and I am reluctant to anyway – turn sex- and gender-based analysis (or any other form of equity analysis) into a set checklist of questions and techniques. To date, my experiences have taught me that these analyses are more fluid and emergent than predictable, and depend on the history of research and evidence in the field, the framing of the question, the sources of data available, whether sex disaggregated data are available, how researchers operationalize the two concepts and the context of the question.” (Respondent #13).
Moving from the analytical framework of sex/gender analysis towards appropriate and robust methods to address these concepts in systematic reviews raised additional challenges. Respondents warned about pitfalls when developing appropriate methods to integrate sex/gender in systematic reviews. In particular, they recognized that subgroup analyses are an important tool for investigating heterogeneity and examining results for men and women separately, but such analyses raise important technical and interpretive challenges:
“Heterogeneous outcomes and small studies make robust subgroup analysis difficult to complete.” (Respondent #3).
“Few primary studies pre-specify subgroup analyses or acknowledge the limitations of analyses as hypothesis-generating. Over-interpretation of results (emphasis on the subgroup analysis rather than the primary outcome) and subsequent claims also present challenges when the emphasis is on subgroup analysis rather than the primary outcome, and give rise to potential for publication bias, and spurious (not to mention some notorious…) subgroup claims in the literature.” (Respondent #21).
On the other hand, respondents considered it important to address the implications of potential sex/gender-based differences and equity issues in relation to health conditions and interventions:
“Reviewers may not recognize that even small differences in the magnitude of treatment effect could translate into meaningful differences to consider in risk-benefit ratios.” (Respondent #21).
Respondents acknowledged that methodological challenges, such as quality of data reporting and appropriate subgroup analyses, were not unique to sex/gender considerations but were of general concern to systematic reviewers. However, they identified a specific need for dialogue and guidance on the application of sex/gender analysis to systematic review methods:
“[There is a] need for appropriate guidance and training for both trialists and systematic reviewers and for leadership in methods, and examples of applying these methods.” (Respondent #17).
Challenges related to availability and quality of data
Linked to the methodological issues were challenges related to the availability and quality of data needed to conduct a robust review: these challenges became particularly marked when matters of sex/gender were considered. Many commented on the lack of data about women, lack of sex-disaggregated data, and/or inconsistencies in reporting such data:
“[There is a] lack of sex-disaggregated data in published studies. Lack of transparency in published studies, for example, cases where some outcomes are reported for both men and women and others are not, without explanation…. [I was] working on a review that examined a health issue specifically among women. It was challenging to find data on women to conduct the review. This stimulated questions around quality, the questions researchers ask and the answers that result from these very specific questions.” (Respondent #3).
“Original studies do not include the kind of disaggregated data that might allow secondary sex/gender analyses to be carried out.” (Respondent #12).
The absence of evidence or the use of suboptimal data were regarded as issues of data quality that affect the conduct of systematic reviews generally. These were seen to stem from “…lack of transparency, access to all the studies, all measured outcomes in studies and to the original data that might allow additional questions to be answered” (Respondent #12). Respondents also pointed to the potential for serious consequences from the absence, omission or other shortcomings in data gathering, analysis, and reporting, such as failure to include data about adverse events.
“A ‘thorny’ issue that deserves special mention is how to find, adequately assess, and incorporate harms data into intervention systematic reviews. This is a consistent problem due to poor quality reporting and/or measurement, suppression of data or delays in publication, short duration of follow-up, and a lack of representation of potentially vulnerable populations for whom the intervention is ultimately targeted.” (Respondent #21).
In particular, if questions regarding sex and gender were not asked, either when conducting single trials or systematic reviews, then the data would not be available to answer questions about possible differences in treatment effects for women or men.
“I think there is an assumption by some, that if the treatment works, why wouldn't it work for everybody? And for those convinced of the importance, many will consider it a great effort to try to obtain this additional data from the authors of the original studies.” (Respondent #19).
In essence, the non-availability and/or non-reporting of data created an iterative cycle in which questions about sex/gender were repeatedly left out. Respondents aptly summarized this issue:
“Questions not asked and therefore not answered.” (Respondent #3).
“…the questions that are asked are often driven by the data that are available [so that] there is some resistance to asking sex/gender/diversity questions.” (Respondent #13).
These challenges underscore the importance of systematic reviews in driving reporting standards for primary studies. Emerging guidance on the use of sex/gender analysis in systematic reviews encourages authors to report what is known and not known about effectiveness of the intervention reviewed [49].
Practical and policy challenges
In addition to the difficulties of obtaining sex-disaggregated data, and a lack of tools, respondents identified other practical challenges to integrating sex/gender in systematic reviews in health. These included an absence of ‘model’ reviews to illustrate how sex/gender analysis could be carried out:
“There are still relatively few examples of solid Cochrane reviews that incorporate a consistent approach to sex and gender analysis and equity analysis.” (Respondent #20).
Other practical challenges dealt with the realities of working in a research intensive environment with competing demands and expanding workloads. Respondents cited experiences of resistance or ‘push back’ to incorporating considerations of sex, gender, and equity in systematic reviews “…from authors and review groups who feel overwhelmed with the workload already demanded of them” (Respondent #9). Respondents also discussed realities of publishing, noting “[there is] difficulty finding journals that think this sort of research has merit” (Respondent #7). In order to address these challenges, respondents saw the need for researchers, reviewers and editors to be aware of the substantive and growing body of health evidence addressing sex/gender similarities and differences in relation to health outcomes. A critical mass of evidence was considered integral to support the merit of such studies.
Respondents also identified several policy challenges that were system level and generally fell outside the scope of systematic review practice. Relating to earlier challenges about the availability of data, respondents described a need for policies that would mandate transparency in research and data collection, calling for increased transparency and more complete reporting of methodological information in current trial registries. These issues have important implications for the quality of systematic reviews.
“[There is] a lack of binding guidance on the part of regulatory agencies and funders of clinical trials to collect the data in a pre-specified manner, and to make the data publicly available.” (Respondent #21).
The types of decisions required to address these challenges are typically made and funded by higher levels of government or quasi-governmental bodies that implement government directions into policy. However, systematic reviewers working at the ‘coalface’ of reported study data have an important role to play in advocating for such policy changes.
Proposed outcomes
While respondents described the challenges and thorny issues in detail, no one answered the question, “How have you attempted to address these?” However, respondents did propose a range of outcomes they hoped would emerge from the workshop. These proposed outcomes indicate how some of the identified challenges could be approached and eventually addressed. Proposed outcomes included developing methods guidelines and/or a series of methods papers to provide a framework and practical ways to integrate sex/gender analysis in systematic reviews and in peer review; refining current tools and checklists for the design and appraisal of reviews and/or developing new tools; and establishing cross-disciplinary collaborations through a ‘community of practice’. Respondents sought ways to determine the “threshold of important differences” of sex/gender (Respondent #13), the influence of sex/gender “on the summary effects in a systematic review” (Respondent #1), and “the key ideas to help increase the sex/gender relevance of reviews…the minimum to consider to maximize the impact” (Respondent #8). Through joint efforts, collaborators could share resources, conduct research, including model reviews, and provide educational fora, training and outreach to biomedical and social scientists, practitioners, peer reviewers, journal editors, funders and policy makers interested in the quality and applicability of health evidence. Collaboration might also result in “a strategy to encourage funding agencies and regulatory agencies to require the collection of key data related to issues of sex/gender, equity, and bias in publicly funded trials, full disclosure of study results, and the publication of protocols” (Respondent #21). Many of these suggestions laid the basis for initiatives discussed below.
