Documenting research with transgender and gender diverse people: protocol for an evidence map and thematic analysis
© The Author(s). 2017
Received: 17 August 2016
Accepted: 4 February 2017
Published: 20 February 2017
There is limited information about how transgender, gender diverse, and Two-Spirit (trans) people have been represented and studied by researchers. The objectives of this study are to (1) map and describe trans research in the social sciences, sciences, humanities, health, education, and business, (2) identify evidence gaps and opportunities for more responsible research with trans people, (3) assess the use of text mining for study identification, and (4) increase access to trans research for key stakeholders through the creation of a web-based evidence map.
Study design was informed by community consultations and pilot searches. Eligibility criteria were established to include all original research of any design, including trans people or their health information, and published in English in peer-reviewed journals. A complex electronic search strategy based on relevant concepts in 15 databases was developed to obtain a broad range of results linked to transgender, gender diverse, and Two-Spirit individuals and communities. Searches conducted in early 2015 resulted in 25,242 references after removal of duplicates. Based on the number of references, resources, and an objective to capture upwards of 90% of the existing literature, this study is a good candidate for text mining using Latent Dirichlet Allocation to improve efficiency of the screening process. The following information will be collected for evidence mapping: study topic, study design, methods and data sources, recruitment strategies, sample size, sample demographics, researcher name and affiliation, country where research was conducted, funding source, and year of publication.
The proposed research incorporates an extensive search strategy, text mining, and evidence map; it therefore has the potential to build on knowledge in several fields. Review results will increase awareness of existing trans research, identify evidence gaps, and inform strategic research prioritization. Publishing the map online will improve access to research for key stakeholders including community members, policy makers, and healthcare providers. This study will also contribute to knowledge in the area of text mining for study identification by providing an example of how semi-automation performs for screening on title and abstract and on full text.
KeywordsEvidence map Transgender Gender diverse Text mining Research ethics Responsible research Research prioritization
The aim of this review is to map and describe how transgender, gender diverse, and Two-Spirit (trans) people have been studied and represented within and across research in the fields of social sciences, sciences, humanities, health, education, and business. There is limited information about the scope of research focusing on trans individuals and communities. Because many people are not aware of the amount of research that has been conducted, this leads to misunderstandings and miscommunication. These beliefs are highlighted in statements by researchers such as “Limited empirical data are available regarding the mental health and general well-being of the transgender population” , “There is a dearth of health research about transgender people” , and “Literature regarding the gender variant population is very limited” . Such misunderstandings may be particularly troublesome if trans community members are unaware of research that can potentially inform questions they have about their lives. Despite the lack of specific information, both researchers and community members have highlighted the links between research and the oppression of trans people [4–6]. Systematic research documenting the types of studies that have been conducted over time will provide details about the evidence that does exist and will help to identify opportunities for more responsible research  with gender diverse individuals and communities.
There are multiple challenges that restrict our ability to conduct reviews in the area of trans research. The first relates to the terminology used to describe transgender, gender diverse, and Two-Spirit people and the ways this influences search strategies. Language used to describe gender diverse people varies across stakeholder communities including medical diagnoses, terms used within or by communities, and phrases used across cultures and linguistic groups. As this terminology evolves over time , it adds to the number of terms that should be included in strong search strategies. A second challenge relates to subject headings, both in terms of the ways these headings reflect trans experience and their inability to remain up to date with language related to gender diversity . These complications necessitate searches beyond subject headings, a process that is made more complex because it is difficult to search terms such as “trans” or “gender identity” by themselves due to the lack of specificity of these terms to the target study records and the consequent number of irrelevant results this produces. It is also necessary for search strategies to include both database-specific headings and independent search terms and to include terms such as mastectomy or vaginoplasty that may be relevant to both cisgender and transgender experience. The term cisgender refers to people who identify with the gender they were labelled at birth, also referred to as non-transgender people. Once searches are complete, screening is complicated by difficulties with identifying whether there are trans participants involved in the studies, or whether the articles are trans-focused, due to information that may be incomplete in the title and abstract. For example, these challenges arise when reviewing references that include trans people as part of larger studies with lesbian, gay, bisexual, trans, and queer (LGBTQ) communities, and surgery-related case reports.
Despite these difficulties, some researchers have attempted to raise awareness of the types of trans research available. One of the earliest examples is an annotated bibliography developed by Denny in 1994 . Published in book format, this bibliography includes early articles, books, and community reports. Since then, we have also seen a slow increase in systematic reviews. Primarily focused in the area of trans health , researchers have conducted reviews related to gender dysphoria , HIV , cancer care , mental health , learning disabilities , support experiences and attitudes of parents of gender variant children , gender identity disorder in twins , and aging . More commonly, we see trans studies included as part of larger reviews focusing on LGBTQ communities, men who have sex with men (MSM), or other marginalized populations (e.g., [20, 21]).
The proposed research, by incorporating an extensive search strategy, text mining, and evidence map, has the potential to build on knowledge in several fields. At this time, there are no evidence maps of trans research. By documenting this broad field of study, this review will increase awareness of existing trans research, identify evidence gaps, and inform strategic research prioritization . Publishing the map online will also improve access to research for key stakeholders including community members, policy makers, and healthcare providers.
Aim and objectives
Document trans research in the fields of social sciences, sciences, humanities, health, education, and business including information about study topic, sample demographics, and study design
Identify evidence gaps and opportunities for more responsible research with trans people
Assess the use of text mining for study identification
Increase access to trans research for community members, policy makers, and healthcare providers by establishing a web-based evidence map, including a searchable reference database.
This protocol was prepared in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P)  (see Additional file 1). An evidence map will be developed using the framework developed by Hetrick and colleagues  which includes four steps. Evidence maps are an emerging method  to “collate, describe, and catalog” knowledge across a broad subject area . This information can then be leveraged by stakeholders to inform policy and clinical decisions .
As part of the process of developing evidence maps, it is recommended that researchers clarify concepts and engage key stakeholders in considering the potential scope of the review . Accordingly, individual consultations were held with members of trans and cisgender communities to discuss terminology, search scope, and potential uses of an evidence map. Based on the results of consultations and pilot searches, the eligibility criteria were established to include all original research studies of any design, reported in English language peer-reviewed journals, that identifiably included trans people or their information, such as medical or surgical case reports with single participants, trans-focused qualitative or quantitative research, and population survey data that adequately identify trans or gender diverse participation.
Initial identification of potential databases was based on the goal of obtaining the broadest range of studies about trans people from multiple fields including social sciences, sciences, humanities, health, education, and business. A secondary emphasis was to gather research from countries and cultures around the world. For example, in order to properly capture research about gender diverse Indigenous people, three databases focused on Indigenous and First Nations research were included.
Once a draft list had been identified, overlap analysis of potential databases was conducted by a health sciences librarian [28–30]. Specifically, PubMed was chosen to capture the content not included in MEDLINE through Scopus . Fifteen databases were selected to ensure the identification of diverse study designs  including Academic Search Premier, Anthropology Plus, Bibliography of Native North Americans, CINAHL, First Nations Periodical Index, Indigenous Studies Portal, LILACS, ProQuest Social Sciences Premium (contains Sociological Abstracts, ERIC, Social Services Abstracts & Applied Social Sciences Index and Abstracts), PsycINFO, PubMed, SciELO, Scopus, Social Work Abstracts, Web of Science, and Women’s Studies International.
Search terms focus on transgender, gender diverse, and Two-Spirit identities and experiences. The search strategy is provided in Additional file 2. Because there are multiple terms used for (and/or by) trans people, and this language continues to shift over time , the full list of search terms is extensive and consists of terms related to gender identity (e.g., “trans woman”), diagnoses (e.g., “gender identity disorder” and “gender dysphoria”), medical and surgical procedures (e.g., vaginoplasty), terms used in a range of countries and cultures (e.g., hijra, waria, travesti), and language used historically (e.g., “transvestite”).
Results of database searches
Academic Search Premier
Bibliography of Native North Americans
First Nations Periodical Index
Indigenous Studies Portal
ProQuest Social Sciences Premium
ProQuest Subject Terms
Social Work Abstracts
Web of Science
Women’s Studies International
Total number of references retrieved
Total number of references
Abstracts will initially be screened based on the information in the title and abstract (level 1). References will be excluded if articles are not written in English, if they are not original research, if they do not include humans, or if they include only cisgender heterosexual people or people diagnosed with disorders of sex development (DSD). If a reference cannot be excluded at level 1, the full text of the article will be uploaded so that it can be screened more thoroughly (level 2).
The large number of citations retrieved by electronic searches in such a complex and broad topic area inevitably creates workload challenges for reviewers who need to check them all for eligibility. The use of new technologies—text mining and machine learning—have been advanced as potential ways in which screening workload might be reduced . When used in the context of reference screening in systematic reviews, a process known as “active learning” can be employed, whereby the machine “learns” from a relatively small sample of reviewer decisions and presents to the reviewer a set of references to screen next; the machine then learns from these screened references too, and the process continues in an iterative fashion. While effective at identifying the majority of relevant studies much earlier in the screening process than would otherwise be the case, there is a danger of the machine models becoming “over-fitted” early in the process, and some relevant studies not being identified. In order to reduce this risk, the citations are grouped together into thematically similar topics using topic modeling using Latent Dirichlet Allocation ; these topics can then be utilized as “features” within the machine learning process and also examined manually by reviewers in order to ensure that each topic has been adequately explored for potentially relevant studies.
Screening on full text
For full-text screening, two team members will review each reference, and any differences will be reconciled through discussion. Level 2 screening will identify original research that includes trans participants or their information. In addition, at this level we will identify studies that include only trans participants, research with photographs of trans people, research that includes trans participants as part of larger LGBTQ studies, and studies with both cisgender and trans participants. The purpose of identifying these details at level 2 is to support data extraction. After eligibility is confirmed based on a review of the full text, then the extraction of information from each article will begin (level 3).
Data collection process
Once all of the English-language peer-reviewed original research that includes trans people or their information has been identified, we will begin data extraction using a standardized data extraction form. The form will be piloted by two reviewers and then data extraction will be conducted by one person, with a second reviewer verifying data extraction results.
Data extraction will focus on creating an evidence map emphasizing the extent and distribution  of trans research studies. The following information will be collected for mapping: study topic; study design, methods, and data sources; recruitment strategies; sample size and demographics (gender identity, sexual identity, race/ethnicity, age, geographic location, education, and income); terminology used to describe trans people; researcher name and affiliation; geographic location of data collection; funding source; and year of publication. Because we do not extract health-related outcomes, this evidence map has not been registered with PROSPERO.
In their recent systematic review, Miake-Lye et al.  highlighted the user-friendly formats of evidence maps, which often include graphs, visual figures, or a database that is searchable. For example, McCandless and Perkins  created an interactive infographic looking at the evidence for nutritional supplements. In addition, researchers including Snilstveit and colleagues  are contributing to gap maps that visually illustrate both evidence and gaps in research. With this project, the goal is to focus on mapping the information stakeholders are most interested in obtaining such as subject area, study design, and sample demographics. After data extraction is complete, information will be exported from EPPI-Reviewer into a database hosted by RSpace Repository at Renison University College, University of Waterloo (http://rspace.uwaterloo.ca/xmlui/). The initial plan is to incorporate an open access searchable database including title, abstract, and journal details, as well as information extracted as part of this evidence mapping process. Once the database has been populated, we will develop additional visually accessible tools that are more accessible to policy makers and community stakeholders, including the ability to combine searches using visual symbols, and to display information using formats such as bubble plots and color-coded summary tables.
This research will map and describe how trans people have been represented and studied within and across multiple fields of research. In addition to identifying the types of research that have been conducted, it will also provide information about which topics have been under-researched, who has been over- or under-included as research participants, and areas where further scoping studies or systematic reviews would be appropriate. Providing this information online will help to improve stakeholder access to research about gender diverse people and will contribute to increased knowledge democracy for transgender, gender diverse, and Two-Spirit individuals and communities. This study will also increase knowledge in the area of text mining for study identification by providing an example of how semi-automation performs for screening on title and abstract and on full text.
Disorders of sex development
Human immunodeficiency virus
Lesbian, gay, bisexual, trans, and queer
Men who have sex with men
Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols
Transgender, gender diverse, and Two-Spirit
The authors wish to acknowledge the support of community members who provided consultation on search terms and data extraction. In addition, Stephanie Power, Christopher Cumby, and Daze Jefferies provided research assistance at different stages of the review process.
During the development of this protocol, ZM was supported by a doctoral fellowship funded by the Canadian Institutes of Health Research and Research and Development Corporation of Newfoundland and Labrador and by the Canadian Mental Health Association-Newfoundland and Labrador. VW holds an Ontario Early Researcher Award. These funders have had no role in developing the protocol for this review.
Availability of data and materials
The datasets extracted and/or analyzed during the current study are available from the corresponding author on reasonable request.
ZM wrote the initial draft of the protocol and is the guarantor of the review. VW provided methodological guidance and revisions to the manuscript. MS assisted in the identification of databases and reviewed the search strategy. CK and FB are co-supervisors of this project. They provided consultation at all stages of review development and contributed revisions to the manuscript. JT and IS are supporting the use of text mining and contributed to the data collection and synthesis sections of the protocol. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Sjoberg MD, Walch SE, Stanny CJ. Development and initial psychometric evaluation of the Transgender Adaptation and Integration Measure (TG AIM). Int J Transgend. 2006;9:35–45.View ArticleGoogle Scholar
- Reisner SL, Gamarel KE, Dunham E, Hopwood R, Hwahng S. Female-to-male transmasculine adult health: a mixed-methods community-based needs assessment. J Am Psychiatr Nurses Assoc. 2013;19:293–303.View ArticlePubMedGoogle Scholar
- Piper J, Mannino M. Identity formation for transsexual individuals in transition: a narrative family therapy model. J GLBT Fam Stud. 2008;4:75–93.View ArticleGoogle Scholar
- Califia P. Sex changes: the politics of trangenderism. San Francisco, CA: Cleis Press; 2003.Google Scholar
- Namaste VK. Invisible lives: the erasure of transsexual and transgendered people. Chicago: University of Chicago Press; 2000.Google Scholar
- Stryker S. Transgender history. Berkeley, CA: Seal Press; 2008.Google Scholar
- Owen R, Macnaghten P, Stilgoe J. Responsible research and innovation: from science in society to science for society, with society. Sci Public Policy. 2012;39:751–60.View ArticleGoogle Scholar
- Reicherzer S. Evolving language and understanding in the historical development of the gender identity disorder diagnosis. J LGBT Issues Couns. 2008;2(4):326–47.View ArticleGoogle Scholar
- Roberto KR. Inflexible bodies: metadata for transgender identities. J Infor Ethics. 2011;20(2):56–64.View ArticleGoogle Scholar
- Denny D. Gender dysphoria: a guide to research. New York, NY: Garland; 1994.Google Scholar
- Reisner SL, Poteat T, Keatley J, Cabral M, Mothopeng T, Dunham E, Holland CE, Max R, Baral SD. Global health burden and needs of transgender populations: a review. Lancet. 2016;388(10042):412–36.View ArticlePubMedGoogle Scholar
- Eftekhar M, Ahmadzad-Asl M, Naserbakht M, Taban M, Jalali A, Alavi K. Bibliographic characteristics and the time course of published studies about gender dysphoria: 1970–2011. Int J Transgend. 2015;16:190–9.View ArticleGoogle Scholar
- Baral SD, Poteat T, Stromdahl S, Wirtz AL, Guadamuz TE, Beyrer C. Worldwide burden of HIV in transgender women: a systematic review and meta-analysis. Lancet Infect Dis. 2013;13(3):214–22.View ArticlePubMedGoogle Scholar
- Watters Y, Harsh J, Corbett C. Cancer care for transgender patients: systematic literature review. Int J Transgend. 2014;15:136–45.View ArticleGoogle Scholar
- Crawford M, Hohn K. Transgender youths: a systematic review of mental health literature. Council on Social Work Education Annual Program Meeting. 2015.Google Scholar
- Wood E, Halder N. Gender disorders in learning disability—a systematic review. Tizard Learning Disability Review. 2014;19(4):158–65.View ArticleGoogle Scholar
- Riggs DW, Due C. Support experiences and attitudes of Australian parents of gender variant children. J Child Fam Stud. 2015;24:1999–2007.View ArticleGoogle Scholar
- Heylens G, De Cuypere G, Zucker KJ, Schelfaut C, Elaut E, Vanden Bossche H, De Baere E, T'Sjoen G. Gender identity disorder in twins: a review of the case report literature. J Sex Med. 2012;9(3):751–7.View ArticlePubMedGoogle Scholar
- Finkenauer S, Sherratt J, Marlow J, Brodey A. When injustice gets old: a systematic review of trans aging. J Gay Lesbian Soc Serv. 2012;24(4):311–30.View ArticleGoogle Scholar
- Collier KL, van Beusekom G, Bos HMW, Sandfort TGM. Sexual orientation and gender identity/expression related peer victimization in adolescence: a systematic review of associated psychosocial and health outcomes. J Sex Res. 2013;50:299–317.View ArticlePubMedPubMed CentralGoogle Scholar
- Lee JGL, Matthews AK, McCullen CA, Melvin CL. Promotion of tobacco use cessation for lesbian, gay, bisexual, and transgender people: a systematic review. Am J Prev Med. 2014;47(6):823–31.View ArticlePubMedPubMed CentralGoogle Scholar
- Snilstveit B, Vojtkova M, Bhavsar A, Gaarder M. Evidence gap maps—a tool for promoting evidence-informed policy and prioritizing future research. 2013. http://documents.worldbank.org/curated/en/2013/12/18648542/evidence-gap-maps-tool-promoting-evidence-informed-policy-prioritizing-future-research.View ArticleGoogle Scholar
- Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart P, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.View ArticlePubMedPubMed CentralGoogle Scholar
- Hetrick SE, Parker AG, Callahan P, Purcell R. Evidence mapping: illustrating an emerging methodology to improve evidence-based practice in youth mental health. J Eval Clin Pract. 2010;16:1025–30.View ArticlePubMedGoogle Scholar
- Miake-Lye IM, Hempel S, Shanman R, Shekelle PG. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev. 2016;5:1.View ArticleGoogle Scholar
- James KL, Randall NP, Haddaway NR. A methodology for systematic mapping in environmental sciences. Environmental Evidence. 2016;5:1.View ArticleGoogle Scholar
- Bragge P, Clavisi O, Turner T, Tavender E, Collie A, Gruen RL. The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol. 2011;11:92.View ArticlePubMedPubMed CentralGoogle Scholar
- Burnham JF. Scopus database: a review. BMC Central. 2006;3:1.Google Scholar
- Chen X. Overlap between traditional periodical indexes and newer mega indexes. Ser Rev. 2006;32:233–7.View ArticleGoogle Scholar
- Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19:591–603.View ArticlePubMedGoogle Scholar
- Althuis MD, Weed DL, Frankenfeld CL. Evidence-based mapping of design heterogeneity prior to meta-analysis: a systematic review and evidence synthesis. Syst Rev. 2014;3:80.View ArticlePubMedPubMed CentralGoogle Scholar
- O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5.View ArticlePubMedPubMed CentralGoogle Scholar
- Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. J Mach Learn Res. 2003;3:993–1022.
- Katz DL, Williams AL, Girard C, Goodman J. The evidence base for complementary and alternative medicine: methods of evidence mapping with application to CAM. Altern Ther Health Med. 2003;9:22–30.PubMedGoogle Scholar
- McCandless D, Perkins A. Snake oil? Scientific evidence for health supplements. 2014. http://www.informationisbeautiful.net/visualizations/snake-oil-supplements/.Google Scholar