Skip to main content

Data quality and feasibility of the Experience Sampling Method across the spectrum of severe psychiatric disorders: a protocol for a systematic review and meta-analysis



Due to a number of methodological advantages and theoretical considerations, more and more studies in clinical psychology research employ the Experience Sampling Method (ESM) as a data collection technique. Despite this growing interest, the absence of methodological guidelines related to the use of ESM has resulted in a large heterogeneity of designs while the potential effects of the design itself on the response behavior of the participants remain unknown. The objectives of this systematic review are to investigate the associations between the design characteristics and the data quality and feasibility of studies relying on ESM in severe psychiatric disorders.


We will search for all published studies using ambulatory assessment with patients suffering from major depressive disorder, bipolar disorder, and psychotic disorder or individuals at high risk for these disorders. Electronic database searches will be performed in PubMed and Web of Science with no restriction on the publication date. Two reviewers will independently screen original studies in a title/abstract phase and a full-text phase based on the inclusion criteria. The information related to the design and sample characteristics, data quality, and feasibility will be extracted. We will provide results in terms of a descriptive synthesis, and when applicable, a meta-analysis of the findings will be conducted.


Our results will attempt to highlight how the feasibility and data quality of ambulatory assessment might be related to the methodological characteristics of the study designs in severe psychiatric disorders. We will discuss these associations in different subsamples if sufficient data are available and will examine limitations in the reporting of the methods of ambulatory studies in the current literature.

Systematic review registration

The protocol for this systematic review was registered on PROSPERO (PROSPERO 2017:CRD42017060322) and is available in full on the University of York website (

Peer Review reports


Several reasons can explain the growing interest of clinical research in ambulatory assessment methods. Patients are generally poor in providing accurate, global, retrospective reflections of real-life experiences and behaviors, which is especially problematic for depressed patients who suffer from cognitive impairment [1] and for psychotic patients who also experience a loss of reality testing [2]. Therefore, the symptomatic dynamics in patients, which are essential for improving the understanding of severe psychiatric disorders [3,4,5,6], cannot be captured precisely through retrospective assessments [7]. Moreover, symptoms occur naturally in the context of daily life and as such, assessing patients in a controlled environment might neglect the influence of context on a patient’s state (e.g., [8, 9]). In this perspective, a growing body of research is now relying on ambulatory assessment methods such as the Experience Sampling Method (ESM, [10]) or Ecological Momentary Assessment (EMA, [11]). Based on repeated and momentary self-evaluations in the individual’s life context, this methodology allows capturing the evolution of affects, cognitions, or behaviors over time with a high ecological validity, as it limits potential artifacts from laboratory settings or retrospective recall [7, 12,13,14].

Despite the rapid increase of the use of momentary assessment methods in clinical psychology and psychiatric research, methodological challenges related to the use of such methods remain. In particular, as this method repeatedly prompts individuals to rate their current affective and cognitive state in the context of their daily life, it is plausible that the repetition of self-evaluations induces reactivity (e.g., increased burden, fatigue, or awareness of cognitive and affective state). A number of studies have reported reactive effects to the repeated assessments in several populations including healthy participants [15, 16] and in individuals suffering from chronic pain [17], substance use disorders [18,19,20,21], depression [22], and anxiety disorder or psychosis [19]. These reactive effects could involve a loss engagement in the participants, decreasing both the feasibility and the data quality in studies relying on this assessment method. Yet, the factors influencing this reactivity have not been clearly defined. In patients suffering from chronic pain, the systematic review of Morren et al. [23] reported several associations between reactivity and (i) methodological parameters related to the design of ESM studies such as the length of the ESM questionnaire or the use of a financial compensation and (ii) the characteristics of the sample such as the age of the participants. In this view, methodological parameters related to the design of ESM studies or to the characteristics of the sampled population are likely to undermine the feasibility and the quality of the data collected. However, the systematic review of Morren et al. [23] focused on momentary assessments related to chronic pain and these observations are likely to differ when targeting different clinical populations. Particularly, this important topic has not been investigated in severe psychiatric disorders while numerous ESM-based studies have been performed in these clinical populations. Considering the heterogeneity of designs in the related literature (e.g., with respect to the duration of the follow-up, the amount of self-evaluations, or the type of device used for data collection), there is a crucial need to answer these methodological concerns.

Our objective is to investigate protocol- and subject-related factors that could undermine the feasibility of ambulatory studies and the quality of the data collected through this methodological framework in psychotic disorder (PD), major depressive disorder (MDD), bipolar disorder (BD), and high-risk individuals for these disorders (HR).


This protocol is based on the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) guidelines [24]. For transparency and completeness, a completed PRISMA-P 2015 checklist is provided as an additional file [see Additional file 1]. The protocol has also been registered in the International Prospective Register of Systematic Reviews database (PROSPERO 2017:CRD42017060322).

Search strategy

A systematic literature search will be performed without publication time limit in PubMed and Web of Science. Observational as well as randomized controlled studies will be included. Case studies, case reports, protocols, study designs, and systematic reviews will not be considered. The search strategy was designed to include relevant terms regarding ambulatory and momentary assessments (e.g., “experience sampling method” and “ecological momentary assessment”) as well as terms related to the clinical diagnosis of the participants under study (e.g., “psychotic disorder”, “major depressive disorder”, “bipolar disorder”). The search strategy will use either MeSH or keyword heading. A concept plan was built with the identified keywords and descriptors to run the search [see Additional file 2]. Unpublished studies as well as studies published in another language than English will not be considered.

Only studies using ESM/EMA designs in PD, MDD, BD, and HR will be included in this systematic review. If available within the included studies, data from non-psychopathological/healthy control groups (HC) will be considered and will serve as a reference group. Ambulatory studies involving only a single daily assessment will be excluded as this time sampling is not representative of the repeated momentary assessment that defines ESM research. To determine the eligibility of the original studies, two researchers (HV and AR) will independently conduct the screening of the studies in the title/abstract and full-text phases based on the inclusion and exclusion criteria. The screening results will be compared in order to identify any discrepancies. In case of a disagreement, a third researcher (IMG) will be consulted. We will use the PRISMA flow diagram to report quantitative and qualitative information on every phase of the selection process.

Data extraction

When available, data will be extracted for the following items: (i) general study characteristics (i.e., authors, title, year, study design); (ii) sample characteristics (i.e., number of participants included in the analysis, age, gender, clinical status, ethnicity, educational status, employment status, marital status, cohabiting status, medication); (iii) ambulatory design characteristics (i.e., number of momentary assessments per day, number of ambulatory assessment days, number of ambulatory assessment periods [continuous or intermittent assessment], delay between ambulatory assessment periods, ambulatory sampling method [fixed, semi-random or random sampling], time intervals between the ambulatory assessments within a day, time intervals between the first and the last ambulatory assessment within a day, time of the start and the end of the ambulatory assessments within a day, number of items in the ambulatory questionnaire, approximate mean duration of the ambulatory questionnaire, type of scales used in the ambulatory questionnaire, type of device used to perform the ambulatory assessment, type of incentive, amount of the incentive); and (iv) the attrition rate (proportion of individuals excluded from the study compared to the number of individuals included at baseline) and the compliance rate (proportion of self-evaluations completed by the participants compared to the theoretical maximal number of self-evaluations allowed by the design) of the included studies. When needed, the corresponding authors of the original studies will be contacted for further information. Data from the included studies will be extracted and stored in a customized spreadsheet structured according to the items mentioned above and that will be provided as part of the supplementary materials in the upcoming systematic review.

Assessment of methodological quality and risk of bias

The current systematic review does not investigate the main outcomes of the studies that will be included. As such, the conventional assessment of methodological quality is not directly applicable. Instead, the quality of the reporting of the attrition and compliance rates in the included studies will be assessed independently by HV and AR. The scale and the scoring developed to perform this assessment will be derived from the four-point item of the Grades of Recommendation, Assessment, Development, and Evaluation Working Group [25] to fit the aims of this review. This scale will be defined clearly and provided as supplementary material in the upcoming systematic review. If needed, the corresponding authors of the original studies will be contacted for further information.

Data analysis

Studies providing sufficient data regarding the main outcomes (i.e., attrition rates and compliance rates) will be analyzed. If data are not amendable to a statistical analysis, a qualitative synthesis of the main outcomes under study will be conducted. If a sufficient amount of commensurable data is available, a quantitative synthesis (meta-analysis) will be performed.

For attrition, we will analyze the reported/calculated proportions. For compliance, there is in principle a proportion (of completed self-evaluations) per participant, but this information is unlikely to be available. Instead, we will analyze the mean proportions (i.e., \( {\overline{p}}_i=\left(\sum \limits_{j=1}^{n_i}{p}_{ij}\right)/{n}_i, \) where p ij denotes the proportion of completed evaluations for the jth participant in the ith study and n i the group size). We expect either \( {\overline{p}}_i \) to be reported directly (either in terms of a proportion or percentage) or the total number of self-evaluations collected, which is easily converted to \( {\overline{p}}_i \) (i.e., if \( {\overline{p}}_i={x}_i/\left({n}_i\times {m}_i\right) \), where x i denotes the total number of self-evaluations collected and m i the theoretical maximal number of self-evaluations allowed by the design).

Both outcomes will be analyzed using a multilevel random-effects model [26] with random effects/intercepts for studies and groups within studies. Also, we will include the group type (factor with five levels: PD, MDD, BD, HR, HC) and differences in protocols as fixed effects in the model. The a priori planned analyses consist of (1) a comparison of groups with different clinical diagnoses (PD vs. MDD vs. BD vs. HR vs. HC), (2) a comparison of population groups (i.e., based on socio-demographic factors), and (3) the influence of protocol characteristics on the outcomes (e.g., low- vs. high-frequency ambulatory sampling in PD). Results will be reported in terms of 95% confidence intervals and (two-sided) p values for each outcome. Heterogeneity will be assessed using the Q test and the I-squared statistic. A qualitative synthesis summarizing the quality of the report of the main outcomes in the database will be provided.

Discussion and conclusion

To date, little to no information is available regarding the methodological factors that could undermine the data quality and feasibility of ambulatory studies related to severe psychiatric disorders despite a large body of research relying on heterogeneous protocols. Assessing this question within an individual study would involve large samples following numerous protocols with varying parameters and as such would be particularly challenging. Accordingly, there is a need to conduct a meta-analysis to provide an exhaustive assessment of the available evidence using clear and reproducible methods. The objective of this systematic review is to synthesize knowledge surrounding the data quality of ambulatory designs in severe psychiatric disorders with respect to various methodological parameters and as a function of the participants’ clinical status. This systematic review aims to provide evidence-based information to researchers and clinicians on the design of ambulatory protocols and, if applicable, to propose guidelines in order to improve the quality of the data collected in this context.



Bipolar disorder


Experience Sampling Method


Healthy controls


High-risk for severe psychiatric disorders


Major depressive disorder


Psychotic disorder


Preferred Reporting Items for Systematic Reviews and Meta-Analyses


International Prospective Register of Systematic Reviews


  1. Edwards L, Pearce S, Collett B-J, Pugh R. Selective memory for sensory and affective information in chronic pain and depression. Br J Clin Psychol. 1992;31(2):239–48.

    Article  PubMed  Google Scholar 

  2. Blum LH, Vakhrusheva J, Saperstein A, Khan S, Chang RW, Hansen MC, et al. Depressed mood in individuals with schizophrenia: a comparison of retrospective and real-time measures. Psychiatry Res. 2015;227(2–3):318–23.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Houben M, Wim VDN, Kuppens P. The relation between short-term emotion dynamics and psychological well-being: a meta-analysis. Psychol Bull. 2015;141(4):901–30. Accessed 1 Dec 2017.

  4. Knowles R, Tai S, Jones SH, Highfield J, Morriss R, Bentall RP. Stability of self-esteem in bipolar disorder: comparisons among remitted bipolar patients, remitted unipolar patients and healthy controls1. Bipolar Disord. 2007;9(5):490–5.

    Article  PubMed  Google Scholar 

  5. Marwaha S, Broome MR, Bebbington PE, Kuipers E, Freeman D. Mood instability and psychosis: analyses of British National Survey Data. Schizophr Bull. 2014;40(2):269–77.

    Article  PubMed  Google Scholar 

  6. Thompson RJ, Berenbaum H, Bredemeier K. Cross-sectional and longitudinal relations between affective instability and depression. J Affect Disord. 2011;130(1–2):53–9.

    Article  PubMed  Google Scholar 

  7. Solhan MB, Trull TJ, Jahng S, Wood PK. Clinical assessment of affective instability: comparing EMA indices, questionnaire reports, and retrospective recall. Psychol Assess. 2009;21(3):425–36.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Myin-Germeys I, Nicolson NA, Delespaul PA. The context of delusional experiences in the daily life of patients with schizophrenia. Psychol Med. 2001;31(3):489–98.

    Article  CAS  PubMed  Google Scholar 

  9. Hershenberg R, Mavandadi S, Wright E, Thase M. Anhedonia in the daily lives of depressed veterans: a pilot report on experiential avoidance as a moderator of emotional reactivity. J Affect Disord. 2017;208:414–7.

    Article  PubMed  Google Scholar 

  10. Larson R, Csikszentmihalyi M. The experience sampling method. In: Reid HJ, editor. New directions for methodology of social and behavioral science. San Francisco: Joey Bass; 1983. p. 41–56.

    Google Scholar 

  11. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol. 2008;4(1):1–32.

    Article  PubMed  Google Scholar 

  12. Ben-Zeev D, Young MA, Madsen JW. Retrospective recall of affect in clinically depressed individuals and controls. Cogn Emot. 2009;23(5):1021–40.

    Article  Google Scholar 

  13. Broderick JE, Vikingstad G. Frequent assessment of negative symptoms does not induce depressed mood. J Clin Psychol Med Settings. 2008;15(4):296–300.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Fredrickson BL. Extracting meaning from past affective experiences: the importance of peaks, ends, and specific emotions. Cogn Emot. 2000;14:577–606.

    Article  Google Scholar 

  15. Courvoisier DS, Eid M, Lischetzke T. Compliance to a cell phone-based ecological momentary assessment study: the effect of time and personality characteristics. Psychol Assess. 2012;24(3):713.

    Article  PubMed  Google Scholar 

  16. Fuller-Tyszkiewicz M, Skouteris H, Richardson B, Blore J, Holmes M, Mills J. Does the burden of the experience sampling method undermine data quality in state body image research? Body Image. 2013;10(4):607–13.

    Article  PubMed  Google Scholar 

  17. Cruise CE, Broderick J, Porter L, Kaell A, Stone AA. Reactive effects of diary self-assessment in chronic pain patients. Pain. 1996;67(2):253–8.

    Article  CAS  PubMed  Google Scholar 

  18. Litt MD, Cooney NL, Morse P. Ecological momentary assessment (EMA) with treated alcoholics: methodological problems and potential solutions. Health Psychol. 1998;17(1):48.

    Article  CAS  PubMed  Google Scholar 

  19. Johnson EI, Grondin O, Barrault M, Faytout M, Helbig S, Husky M, et al. Computerized ambulatory monitoring in psychiatry: a multi-site collaborative study of acceptability, compliance, and reactivity. Int J Methods Psychiatr Res. 2009;18(1):48–57.

    Article  PubMed  Google Scholar 

  20. Uchalik DC. A comparison of questionnaire and self-monitored reports of alcohol intake in a nonalcoholic population. Addict Behav. 1979;4(4):409–13.

    Article  CAS  PubMed  Google Scholar 

  21. Freedman MJ, Lester KM, McNamara C, Milby JB, Schumacher JE. Cell phones for ecological momentary assessment with cocaine-addicted homeless patients in treatment. J Subst Abus Treat. 2006;30(2):105–11.

    Article  Google Scholar 

  22. Vachon H, Bourbousson M, Deschamps T, Doron J, Bulteau S, Sauvaget A, et al. Repeated self-evaluations may involve familiarization: an exploratory study related to Ecological Momentary Assessment designs in patients with major depressive disorder. Psychiatry Res. 2016;245:99–104.

    Article  PubMed  Google Scholar 

  23. Morren M, Dulmen S, Ouwerkerk J, Bensing J. Compliance with momentary pain measurement using electronic diaries: a systematic review. Eur J Pain. 2009;13(4):354–65.

    Article  PubMed  Google Scholar 

  24. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Andrews JC, Schünemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, et al. GRADE guidelines: 15. Going from evidence to recommendation—determinants of a recommendation’s direction and strength. J Clin Epidemiol. 2013;66(7):726–35.

    Article  PubMed  Google Scholar 

  26. Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis: three-level meta-analysis. Res Synth Methods. 2011;2(1):61–76.

    Article  PubMed  Google Scholar 

Download references


Not applicable.


This research was supported by an ERC consolidator grant to IMG (ERC-2012-StG, project 309767—INTERACT), by an FWO Odysseus fellowship to IMG (G0F8416N), and by the Innovative Medicines Initiative 2 Joint Undertaking (grant number: 115902, RADAR-CNS).

Availability of data and materials

All data generated or analyzed during this study will be included in the upcoming article [and its supplementary information files].

Author information

Authors and Affiliations



All authors participated in the conception of the study and the development of the design of the protocol. HV and AR drafted the manuscript, conducted the search strategy, performed the literature search, and will extract the data and perform the data analysis. WV and IMG will supervise this work. All authors will participate in interpreting the findings. All authors have read and approved the manuscript.

Corresponding author

Correspondence to Hugo Vachon.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) 2015 checklist: recommended items to address in a systematic review protocol. (DOC 55 kb)

Additional file 2:

Concept plan of the search strategy. (DOCX 12 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vachon, H., Rintala, A., Viechtbauer, W. et al. Data quality and feasibility of the Experience Sampling Method across the spectrum of severe psychiatric disorders: a protocol for a systematic review and meta-analysis. Syst Rev 7, 7 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: