We will undertake a realist synthesis. The aim of the study is to establish how, why and in what circumstances networked, inter-organisational IT systems affect patient safety. The objectives of the study are to:
- 1.
Identify initial programme theories and prioritise theories to review;
- 2.
Search systematically for evidence to test and refine the theories;
- 3.
Undertake quality appraisal and use included texts to support, refine or reject programme theories;
- 4.
Synthesise the findings;
- 5.
Disseminate the findings to a range of audiences.
The following sections focus on the first four objectives.
Stage 1: Identification of programme and mid-range theories
This stage is essentially developmental in nature. The first step in this review will be to construct one or more programme theories, concerning the use of networked IT systems and their effects on patient safety. Mid-range theories, which are usefully thought of as a broader class of theory than programme theories, and which will be used to generalise findings at the end of the review, will also be identified. We will supplement our existing knowledge of candidate theories with literature searches to define and develop them. The programme theories will determine the scope (inclusion and exclusion criteria) for the searches and synthesis in the following stages, which will be used to support, refine or reject each theory examined.
Search strategy and information resources
Theories can be explicitly mentioned in research articles and policy documents, or they can be implied in the introductory or discussion sections of documents. They can also be found in commentaries and opinion pieces. We will use conventional literature searching using free text words, synonyms and subject index terms, and some CLUSTER searching techniques (identifying a few key relevant studies and finding further relevant studies via forwards and backwards citation searches, author searches, searching for reports of a particular project) [20]. Using this combined approach, we aim to identify literature that leads us to theories or fragments of theories that can be used to construct programme and mid-range theories.
We anticipate that three searches will supplement the policy documents and academic literature already known to the project team. MEDLINE (1946–present) and EMBASE (1947–present) will be searched as a core set of databases for all of the theory generating searches.
- 1.
Background search of systematic reviews. We will search for systematic reviews that link IT systems and patient safety. This will identify any reviews on this topic that have been published since our scoping review, undertaken before the start of the project. The reviews may describe programme theories or fragments in their introduction or discussion sections, providing insights into the sequences of events linking the intervention and outcomes. We will search the core databases plus the Cochrane Database of Systematic Reviews, the Epistemonikos database and Health Systems Evidence (McMaster University). An example MEDLINE search strategy containing a full set of search terms is available in the Additional file 1 (Search 1).
- 2.
Policies, opinion pieces and research reports. We will search the core databases plus Health Management Information Consortium (1983–present) and Web of Science - Science Citation Index (1990–present) for policy documents, opinion pieces (e.g. editorials) and reports describing leading theories about the relationships between IT systems and patient safety (Additional file 1 Search 2). We will also undertake Google searches to locate reports on key policies, e.g. about the Health Information Technology for Economic and Clinical Health (HITECH) Act 2009, a major US initiative promoting the implementation of cross-organisational IT networks.
- 3.
Author search and Citation search. We will search for reports and articles authored by influential commentators in the core databases plus Health Management Information Consortium (1983–present), Web of Science - Science Citation Index (1990–present), Google Scholar and Scopus (1823–present). Literature by David Bates, the most cited author in the health informatics literature, and Robert Wachter, the author of an influential report on IT in the NHS in England, will be searched (Additional file 1 Search 3). Citation searching may be required due to the iterative nature of developing searches for a realist synthesis.
Inclusion, analysis and synthesis
The records identified in the searches will be saved and managed in an EndNote library. Details of all search activities (databases, websites, date of search, number of records found, search strategies) will be recorded in a timeline spreadsheet. The inclusion criteria for the three searches will be:
IT networks that link two or more organisations outside (but possibly including) hospitals;
IT networks that support direct treatment and care;
Arguments that identify relationships between IT networks and patient safety;
Published in the English language between 2000 and 2018.
Non-English language articles will not be included. The project does not have sufficient resources to translate the full text of articles that may be relevant. In conventional systematic reviews, it is not necessary to translate whole papers, as it is only necessary to identify defined data. In this review, however, we will be looking for data that might occur anywhere in a paper, and a full translation would be needed.
The exclusion criteria will be for studies that:
Describe hospital-only IT systems;
Describe systems that do not link two or more distinct services;
Focus on IT systems that support secondary uses of data, e.g for service planning, research;
are published before 2000;
are published in languages other than English.
Two reviewers (MA and JK) will first independently screen the titles and abstracts of the records for relevance and then assess full-text reports. Any discrepancies will be solved by discussion, and if needed consultation with a third author (JG or RR). Data extraction forms will be developed to capture basic details of studies—authors, publication year, etc.—and passages on theories and theory fragments.
The selected studies will be used to develop visual representations of programme theories, with accompanying text that explains the reasoning that underpins those theories. Experience gained in earlier reviews suggests that there are likely to be several programme theories at this point, and that some of them will be partial, in the sense that the chains of reasoning are ‘high level’ and not fully articulated, or only cover some of the steps linking networked information technologies and patient safety. Where available, claims about the reasons why programmes succeed or fail in practice will be used to annotate the representations.
The programme theories will be used as the basis for consultation with three groups of stakeholders—policy makers, senior IT managers and frontline clinicians. We will use the nominal group technique, which has been used in a previous realist study [21]. At the nominal group meetings participants will be asked to comment critically on the programme theories, on the basis of their knowledge and experience. They will also be asked to develop and then prioritise theories, or particular chains of reasoning within theories, for further study. The prioritisation will take into account the potential to provide learning for the NHS, and the types of networked systems that NHS organisations are implementing. The groups will be re-convened for consultation by email at the end of stage 4 (see below).
The outputs of the three groups will be further reviewed by a patient and public involvement (PPI) group, and discussed with our project steering group. Following these meetings, we will decide on the programme theories, and key elements of those theories, that we will explore in depth in stages 2–4.
Stage 2: Systematic search for evidence
The next stage of the review is a search for empirical studies to test and refine the leading programme theories identified in stage 1. The initial searches will be designed to identify evidence about the steps in the chains of reasoning in each theory. Literatures often focus on one or other section in a chain of reasoning, and as a result, individual searches will often focus on sections rather than a whole programme theory [22]. We will undertake searches using resources which span health and computing literatures including—but not limited to—MEDLINE (1946–present), EMBASE (1947–present), Web of Science Core Collection (1900–present) and INSPEC (1896–present).
We anticipate hand searching papers from leading conferences which are not indexed, for example Software Engineering in Healthcare workshops papers, from the International Conference on Software Engineering. The search strategies for identifying empirical evidence for programme theories can only be fully developed once the programme theories are agreed. However, we anticipate they will contain search concepts for an aspect of patient safety such as medication reconciliation, inter-organisational IT networks and evaluative studies.
The search results will be reviewed in stage 3 (evidence review, see below) and further searches will develop iteratively to follow lines of enquiry. Initial searches may not identify empirical evidence that supports or rejects a programme theory. If that happens, the search will be re-designed to capture empirical evidence that may be found in a different discipline or information resource. For example, we could extend the scope of our searches to look for evidence about other IT applications, including hospital-based systems, and/or extend the scope of the populations of interest. As in stage 1, these searches may use CLUSTER search techniques as an efficient method for finding relevant papers [20].
The results of the electronic searches and all references that are retrieved for stage 2 will be kept in the same EndNote library as those found during stage 1. Details of all search activities during stage 2 will be recorded in the timeline spreadsheet.
Stage 3: Evidence review and quality appraisal
Titles and abstracts of records identified, and the full-text papers selected in stage 2, will be independently screened by two reviewers (MA and JK) to identify those which contain evidence that sheds light on one or more elements of the programme theories identified in stage 1. The RAMESES I guidance states that:
An appraisal of the contribution of any section of data (within a document) should be made on two criteria:
• Relevance – whether it can contribute to theory building and/or testing; and
• Rigour – whether the method used to generate that particular piece of data is credible and trustworthy [14].
We will follow Rycroft-Malone and colleagues in developing criteria for judging relevance [23]. We will, further, use the mid-range theory developed in stage 1 to refine the criteria. Rigour refers to the requirement for an investigation to be of sufficient standard within type, whether that is a process evaluation, an ethnography or other type of study [14].
The empirical data for supporting and/or refuting programme theories will be extracted from the included studies. It is anticipated that a significant proportion of the evidence will be in the form of narrative data and will accordingly be copied into Word files. To maximise accuracy and transparency, a proportion of data extraction will be performed independently by two members of the research team.
Stage 4: Synthesis
Synthesis involves two distinct, but linked, activities. In the first, the empirical evidence identified in stages 2 and 3 will be used to evaluate the programme theories developed in stage 1. In the most straightforward case, the evidence will support the chains of reasoning in one programme theory and serve to reject alternative or competing theories. Less straightforwardly, the evidence might provide support for one part of a programme theory and adverse evidence for another part of the same theory. Or, it might not ‘fit’ a programme theory, neither supporting nor undermining it. Both instances suggest that there may be a problem with the programme theory itself and will lead us to refine it, to achieve a better fit between evidence and theory. On the basis of experience of earlier realist syntheses, we expect that at least one of the selected theories—or theory fragments—will be reasonably well supported by empirical evidence, and at least one will not be not supported.
Second, when a settled programme theory or theories have been produced, they will be interpreted in the broader context of the mid-range theory. This involves abduction, where inferences that lead to the best available explanation are identified. The details of the abductive reasoning processes vary from review to review, but a key point is that it involves inter-play between situation-specific programme theories and broader mid-range theory [24]. In the simplest case, the (now evidence-based) programme theories will be consistent with the mid-range theory. If we find that a programme theory holds across a number of settings (e.g. different combinations of health services and/or different patient groups), this will increase our confidence in it. Alternatively, evidence or argument (or both) may point in different directions, and the wider project team will use the mid-range theory to ‘adjudicate’ between contending programme theories.
Nominal group email consultation
The nominal groups will be re-convened for email consultation. We will summarise our findings to this point, including our provisional syntheses, and present them to the groups. They will be asked to comment on the findings, including whether any of the theories can be rejected, and whether any further searches are merited. The PPI group will also meet at the end of this stage and review the findings and interpretations of the three nominal groups. We will refine our interpretations on the basis of the comments of all four groups.