Tools for assessing the methodological limitations of a QES—a short note

Abstract

The increasing prevalence and application of qualitative evidence syntheses (QES) in decision-making processes underscore the need for robust tools to assess the methodological limitations of a completed QES. This commentary discusses the limitations of three existing tools and presents the authors’ efforts to address this gap. Through a simple comparative analysis, the three tools are examined in terms of their coverage of essential topic areas. The examination finds that existing assessment tools lack comprehensive coverage, clarity, and grounding in qualitative research principles. The authors advocate for the development of a new collaboratively developed evidence-based tool rooted in qualitative methodology and best practice methods. The conclusion emphasizes the necessity of a tool that can provide a comprehensive judgement on the methodological limitations of a QES, addressing the needs of end-users, and ultimately enhancing the trustworthiness of QES findings in decision-making processes.

Background

As qualitative evidence syntheses (QES) are becoming more common and increasingly used in decision-making processes [1,2,3,4,5], there is a need for a tool to assess the methodological limitations of a complete QES. This methodological assessment tool could help users to understand the trust they can place in the findings of a QES and help to interpret further use. In our work, this type of assessment tool would primarily be useful when an existing QES is found that answers a commissioner’s question. In this case, we need to be able to assess the methodological limitations of the completed QES to make a judgement for the commissioner on the extent to which the findings can be trusted and used to suit their purposes. We refer to an assessment tool, and not a checklist, as a deeper methodological understanding of the limitations of a QES is needed to assess the synthesis and how its methodological limitations impact on further use. We believe that the scoring or ranking which are the products of a checklist would not allow for a deep enough evaluation of and reflection around the methodological limitations of the QES and how they relate to the context and question that the QES is going to be used in and for.

The foundation for the discussion in this commentary was a teaching experience the team had in 2022. A request for course content was how to assess the methodological limitations of a QES. We recently had an in-house discussion about the three tools we had identified as options for assessing the methodological limitations of a QES. All three tools are in beta or preliminary versions. We discovered that the assessment tools are not easily accessible. Knowledge of their existence and whereabouts is necessary to locate them. All tools have been developed to meet an internal need (fit for purpose). None of them have been developed through best practice methods [6, 7]. The three tools are:

• Tool 1: Criteria for assessing how well a qualitative evidence syntheses (systematic reviews of qualitative studies) was conducted, a tool developed by Lewin and colleagues in 2012 [8, 9]

• Tool 2: a prototype assessment tool based on AMSTAR 2 [10], Measurement Appraisal Checklist to Assess Qualitative Evidence Syntheses (MACAQuES) by Booth and colleagues from 2019 [11]

• Tool 3: Review template for qualitative evidence synthesis (QES), developed by the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU) based on the ENTREQ reporting guidance [12] in 2023, is published but is still marked as “under development” [13]

We wanted to expand our in-house discussion further for teaching purposes. To do this, first, the authors compared the QES methodological assessment tools in a table and through discussion. Next, we incorporated them into an introductory course on QES methods delivered in October 2022. During the course, we had students reflect over any topics or questions they felt were missing from the existing tools based on the course content. Finally, we reflected on the student feedback and our experiences to assess and conclude that none of the tools fully met our needs.

In this short note, we aim to briefly present and compare items across the three assessment tools we identified and describe what we believe to be their strengths and limitations.

Three assessment tools

We have compared the three QES assessment tools (see Table 1). An x was placed in the table if an item was mentioned in a question or a prompt.

Table 1 A comparison of the topics covered in the three assessment tools

Seven of the eighteen topic areas are covered in all three tools (review question, inclusion criteria, literature search, methodological assessment of the included studies, analysis/synthesis, findings and reflexivity). Four of the topic areas are covered by two tools; a description of the excluded studies is covered in tool 1 and tool 2. Planning/protocol, conflict of interest, and confidence in the findings are covered in tool 2 and tool 3. However, tool 1 was published before the use of GRADE CERQual was implemented, so it is not surprising that that topic area is missing. Six topic areas are covered only by one assessment tool; tool 2 asks users to think through patient involvement, the description of the included studies, data extraction/coding, and dissemination bias. Tool 3 asks users to reflect on researchers’ competence, screening, and other.

Tool 1 has considerably fewer topic areas but includes prompt questions to help the user think through the topic areas. Tool 2 also provides prompt questions or items of note that users should consider when thinking through the topic area. Tool 3 is accompanied by a user guide.

All three tools require experience with and knowledge of qualitative research. This knowledge is needed to interpret the items/questions in a “qualitative manner” to ensure that methodological limitations relevant to qualitative research are assessed. Many questions are not explicitly formulated, meaning that the end user needs to understand qualitative research principles and practices to interpret and apply them. For example, a detailed knowledge around searching and where relevant qualitative evidence is located [13], a knowledge of which synthesis method is appropriate for which type of question [8, 9, 11, 13], and a knowledge of the QES authors background, experience, and competence [13]. Finally, the tools raise concepts that may be new to some researchers such as the concept of the impact of dissemination bias in primary qualitative research and its implications on QES findings [11].

Need for collaboration in developing a new evidence-based methodological assessment tool for QES

Based on our comparison of the three assessment tools, we think there is a need to systematically search for map and assess existing tools. If there is not an existing tool which has been developed in an evidence-based way, then a tool should be considered. Ideally, the end goal would be to develop a new assessment tool that is based on the principles of qualitative research and qualitative evidence syntheses using best practice methods for assessment tool development. The development of the new tool should follow best practice methods so that it reflects all items relevant for the assessment of a completed QES, is based on qualitative methodology, and addresses the needs of the end user—being able to assess the limitations of a completed QES.

This process should be a collaborative effort within the QES community. The first step would be a systematic search for existing tools and the identification of relevant principles in these tools. Additional principles should be gathered from focus groups. This exploratory step would be followed by a Delphi process where stakeholders could come to an agreement on the principles that should be included in a future tool. After the consensus process has been completed, an assessment tool could begin to be developed and user tested.

Recently, this process of collaboratively developing an evidence-based tool for the assessment of the methodological limitations of primary qualitative studies included in a QES (CAMELOT) has been completed [5, 14,15,16]. The CAMELOT project followed the same process we describe above, involving a large number of relevant stakeholders in a collaborative process to determine what was important to include and how the tool could be used. CAMELOT, along with other previous [17,18,19,20,21,22] and ongoing [23] projects that have used the same methodology, lead us to believe that this process would lead to an evidence-based assessment tool for the assessment of the methodological limitations of a QES. We believe that the development of a tool for assessing the methodological limitations of a qualitative evidence synthesis is needed.

Conclusion

In conclusion, we believe that none of the QES methodological assessment tools covered all of the areas that were raised by students as well as our reflections from working in the field. We found that the tools did not seem to be clearly grounded in qualitative research methods (for example words or expressions common in quantitative research were used). We also found that they could not provide a comprehensive/complete judgement on the methodological limitations of a QES that we could present to a commissioner or use to make a decision as critical areas or items were missing that we feel should be considered. We believe that the development of a tool for assessing the methodological limitations of a qualitative evidence synthesis is needed.