Skip Navigation
Skip to contents

JPMPH : Journal of Preventive Medicine and Public Health

OPEN ACCESS
SEARCH
Search
Advanced Search
mobile menu button

Articles

Page Path
HOME > J Prev Med Public Health > Volume 49(2); 2016 > Article
Brief Report
 Citation Discovery Tools for Conducting Adaptive Meta-analyses to Update Systematic Reviews
Jong-Myon Baeorcid, Eun Hee Kimorcid
Journal of Preventive Medicine and Public Health 2016;49(2):129-133.
DOI: https://doi.org/10.3961/jpmph.15.074
Published online: March 14, 2016
  • 11,852 Views
  • 110 Download
  • 19 Crossref
  • 13 Scopus

Department of Preventive Medicine, Jeju National University School of Medicine, Jeju, Korea

Corresponding author: Jong-Myon Bae, MD, PhD  102 Jejudaehak-ro, Jeju 63243, Korea  Tel: +82-64-755-5567, Fax: +82-64-725-2593 E-mail: jmbae@jejunu.ac.kr
• Received: December 24, 2015   • Accepted: March 7, 2016

Copyright © 2016 The Korean Society for Preventive Medicine

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

prev next
Full Article

Download PDF Download PDF

  • Objectives:
    The systematic review (SR) is a research methodology that aims to synthesize related evidence. Updating previously conducted SRs is necessary when new evidence has been produced, but no consensus has yet emerged on the appropriate update methodology. The authors have developed a new SR update method called ‘adaptive meta-analysis’ (AMA) using the ‘cited by’, ‘similar articles’, and ‘related articles’ citation discovery tools in the PubMed and Scopus databases. This study evaluates the usefulness of these citation discovery tools for updating SRs.
  • Methods:
    Lists were constructed by applying the citation discovery tools in the two databases to the articles analyzed by a published SR. The degree of overlap between the lists and distribution of excluded results were evaluated.
  • Results:
    The articles ultimately selected for the SR update meta-analysis were found in the lists obtained from the ‘cited by’ and ‘similar’ tools in PubMed. Most of the selected articles appeared in both the ‘cited by’ lists in Scopus and PubMed. The Scopus ‘related’ tool did not identify the appropriate articles.
  • Conclusions:
    The AMA, which involves using both citation discovery tools in PubMed, and optionally, the ‘related’ tool in Scopus, was found to be useful for updating an SR.
  • Keywords: Meta-analysis, Review, Database, Research design, Computing methodologies
A systematic review (SR) is a research methodology that synthesizes the findings of studies with the same research hypothesis to produce a summary conclusion on the hypothesis [1]. The SR process involves a comprehensive search of related evidence, followed by a meta-analysis, which summarizes the extracted results. The literature search for an SR is conducted through an electronic search, which applies a search algorithm comprised of related keywords related by Boolean logic (AND, OR, NOT) in databases such as PubMed (www.ncbi.nlm.nih.gov/pubmed/) [2]. The publication date of research articles is reflected as a limitation in search conditions because of depending on the timing of the literature search. Furthermore, updates should be conducted regularly to reflect new evidence that has been published after the literature search was conducted for the previous published SR [3,4].
In the 2008 Cochrane Handbook for Systematic Reviews of Interventions, no specific updating methodology is mentioned, although the handbook requires updates to be conducted within two years of the initial publication of the SR [5]. However, this recommendation is rarely followed [6,7]. This is because methodologies for performing the update have not yet been established [8,9], and the update requires human resources and time [4]. For topics in which the number of searching engines is increasing, the burden of updating necessarily increases as well [10]. Therefore, methodologies that enable more efficient searches are needed.
PubMed and Scopus (www.scopus.com), which are the main databases searched for SRs, each have citation discovery tools that provide a list of articles that cited the article of interest and a list of similar or related articles. We developed an SR update methodology that uses these tools. Lists of articles citing or related to the existing SR are first constructed from the citation discovery tools without a separate electronic search, and then articles to be used in the new meta-analysis are selected [11-14]. This methodology assumes that studies conducted with the same research hypothesis have a high likelihood of citing the articles included in the original SR or of being in the list of similar/related articles, and thus that our method will produce a similar list to what the Cochrane review update (CRU) method would produce. In order to differentiate the new methodology from the CRU, which reproduces the SR according to a previously developed protocol, we have named this methodology ‘adaptive meta-analysis’ (AMA) [11-14]. The AMA approach to SR updating was developed on the basis of the techniques for using the citation discovery tools we found to be most appropriate.
The ‘related articles’ citation discovery tool in Scopus produces lists of more than 5000 citations per article, which makes the search too broad to be useful. On the other hand, it is unclear whether the more targeted citation discovery tools that produce shorter lists are inclusive enough. It is worth determining whether using the citation discovery tools of Scopus and PubMed without performing other searches is sufficient to produce a comprehensive list of the articles that should ultimately be included in an SR update. Thus, this study aims to evaluate the value of using the citation discovery tools following the AMA method for an SR update.
In order to investigate the usefulness of the ‘cited by’ citation discovery tool of Scopus, we selected the SR published by one of the present co-authors [15] in 2008 because the SR needed an update and the SR authors had already published a summary of findings published later on the hypothesis covered by the SR [12]. The electronic databases used by Bae et al. [15] for the original SR were PubMed, Ovid MEDLINE, and Embase. The search terms used were ([pancreas] OR [pancreatic]) AND ([neoplasm] OR [cancer]) AND ([fruit] OR [citrus]). For each of 10 articles (nine articles and the SR of Bae et al. [15]), we used the citation discovery tools to produce lists of ‘cited by’ articles provided by PubMed (PM_C), ‘similar articles’ provided by PubMed (PM_S), and ‘cited by’ articles provided by Scopus (SC_C). The long lists of ‘related articles’ provided by Scopus (SC_R) were used just to confirm the number of articles included in the lists. The latest publication date included in the search was late November 2015.
The three types of lists were divided into the following four groups: (A) lists of articles in both PM_C and SC_C, (B) lists of articles in both PM_S and SC_C, (C) lists of articles included only in SC_C, and (D) lists of articles included only in PM_C.
The inclusion criteria for the updated meta-analysis were as follows: analytical epidemiological studies (i.e., case-control studies or cohort studies) that investigated whether pancreatic cancer risk decreased with the level of citrus fruit intake. We evaluated the abstracts and full texts of articles included in the three lists (PM_C, PM_S, SC_C), and then excluded articles according to the following criteria: (1) laboratory studies on animal, cellular, and genetic levels, (2) studies on diseases other than pancreatic cancer, (3) studies that involved developing research methodologies, (4) studies on disease statistics, (5) expert opinions or reviews, (6) systematic reviews, (7) observational studies without information on citrus fruit intake, and (8) clinical studies of patients. According to the aforementioned criteria, we investigated the distribution of articles by the three citation discovery tool lists. The distribution of the selected articles was used as an index for evaluation of the usefulness of the citation discovery tools for the SR update.
When the ‘cited by’, ‘similar articles’, and ‘related articles’ citation discovery tools in PubMed and Scopus were applied to the nine articles selected in the meta-analysis by Bae et al. [15] plus the article by Bae et al. itself, the PM_C lists contained a total of 173 articles, while the PM_S lists included 2380 articles. The SC_C lists contained 573 articles, while the SC_S lists comprised 95 365 articles. The fact that there were approximately 9500 articles in the SC_S list for each article confirmed that the Scopus ‘related articles’ tool produced a search that was too broad to be useful.
Table 1 shows the overlap between the 573 articles in the SC_C lists and the PM_C and PM_S lists. Among the 173 articles in PM_C, 154 (89.1%) articles overlapped with SC_C; among the 2380 articles in PM_S, 92 (3.9%) articles overlapped with SC_C. In other words, among the 573 articles in SC_C, 246 (= 154 + 92) articles were also included in PM_C and PM_S, showing an overlap of 42.9% (= 246 / 573).
Table 2 shows the distribution of the articles according to their characteristics after the aforementioned exclusion criteria were applied. Among the four groups shown in Table 1, the group that contained the highest number of ultimately selected articles was the set of overlapping articles in PM_C and SC_C. In contrast, in the group of articles included only in SC_C, no article was selected for the meta-analysis update.
The finding that no article was selected from the list of articles included only in SC_C, which is also shown in Table 2, signifies that the ‘cited by’ and ‘similar articles’ tools provided by PubMed are sufficient for SR updates. However, since many articles were selected from the list of articles that appeared in both PM_C and SC_C, the efficiency of searches can be increased by using the overlapping articles from the two lists. We recommend this approach as the search strategy for the SR update method we have named the ’AMA’ [11-14].
For CRU, confirmation of the necessity of updates, determination of the timing of updates, and cumulative meta-analysis have been discussed [3,16]. Moreover, additional complications, such as the need to reflect changes in the calculation of risk of bias due to updates, have been suggested [4]. Given that consensus has not yet been reached on the proposed CRU update methodology, we present the AMA as a more feasible alternative to the CRU (Table 3). A CRU involves performing literature searches repeatedly at fixed intervals, using the search algorithms from the previous meta-analysis protocol [5]. In contrast, the AMA method develops a list of articles to analyze based on the articles that were selected in the original SR, which had involved searches using a stated algorithm. AMA compiles an updated list of articles for analysis by producing lists of articles that cite or are similar to the articles included in the SR, using citation discovery tools. Therefore, the most notable difference between the two methods is that CRU values protocols to generate reproducible results while AMA values lists of selected articles to generate acceptable results. Since updating an SR can mean that a meta-analysis must be conducted again on additionally selected articles that are consistent with a certain research hypothesis, we used the term ‘adaptive’ to distinguish our method from simply ‘updating’ the SR.
AMA has several advantages. First, omitting electronic searches reduces the time and effort needed to update an SR. The fact that all articles that need to be selected can be found using only the two citation discovery tools of PubMed is encouraging. Second, AMA is also useful for identifying included articles that report on the same cases or cohort [12]. When the ‘cited by’ and ‘similar articles’ tools are used, duplicates among published articles can be easily assessed so that the list of articles to be including in the meta-analysis can be modified as needed. Third, since the AMA method can confirm errors in previously published SRs and can additionally conduct a sensitivity analysis or subgroup analysis while adding articles that were omitted in the previous SRs, it can be used as a methodology for meta-epidemiology [17].
The validity of AMA remains to be investigated. AMA cannot yet be directly compared to other methods since there is no consensus on the criteria for including articles in a meta-analysis. Nevertheless, when AMA was applied to previous SRs in which a hand search was conducted after an electronic search, we confirmed that several articles that should have been selected, when considering the search period, were omitted [11,13,14]. This indicates that AMA can increase the accuracy in a SR literature search. In the future, it is expected that applying AMA to more studies will show that AMA is useful as a methodology for updating SRs.

CONFLICT OF INTEREST

The authors have no conflicts of interest associated with the material presented in this paper.

Table 1.
Distribution of lists obtained from citation discovery tools in Scopus and PubMed
Groups Scopus ‘cited by’ (n = 573)
[A] PM_C & SC_C Overlapping with PubMed ‘cited by’ (n = 173) 154
[B] PM_S & SC_C Overlapping with PubMed ’similar articles’ (n = 2380) 92
[C] SC_C Only in Scopus ‘cited by’ 327
[D] PM_C Only in PubMed ‘cited by’ 19

PM_C, ‘cited by’ articles provided by PubMed; SC_C, ‘cited by’ articles provided by Scopus; PM_S, ‘similar articles’ provided by PubMed.

Table 2.
Distribution of study characteristics among the 4 groups in Table 1
Study characteristics [A] PM_C & SC_C (n = 154) [B] PM_S & SC_C (n = 92) [C] SC_C (n = 327) [D] PM_C (n = 19)
Laboratory studies 30 0 81 1
Studies of diseases other than pancreatic cancer 7 1 22 1
Methodology-focused studies 0 0 11 0
Studies on disease statistics 4 1 14 1
Expert opinions and reviews 16 13 126 3
Systematic reviews 16 7 15 4
Observational studies without information on citris fruit intake 59 64 31 7
Clinical studies 9 0 27 0
Articles ultimately selected 13 6 0 2

PM_C, ‘cited by’ articles provided by PubMed; SC_C, ‘cited by’ articles provided by Scopus; PM_S, ‘similar articles’ provided by PubMed.

Table 3.
Differences between the Cochrane review update (CRU) and adaptive meta-analysis (AMA) for updating a systematic review (SR)
CRU AMA
Protocol revision Mandatory Flexible
Starting point Study protocol Articles selected by previous SR
Database search Mandatory If needed
Search method Search algorithm Citation discovery tools
Update intervals Regular Any time
Update authors Original authors Anyone
Study design SR SR with meta-epidemiology1

1 Bae [17] describes the concept of ‘meta-epidemiology’.

  • 1. Tseng TY, Dahm P, Poolman RW, Preminger GM, Canales BJ, Montori VM. How to use a systematic literature review and meta-analysis. J Urol 2008;180(4):1249-1256ArticlePubMed
  • 2. Engberg S. Systematic reviews and meta-analysis: studies of studies. J Wound Ostomy Continence Nurs 2008;35(3):258-265ArticlePubMed
  • 3. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. Considerations from the risk of bias perspective for updating Cochrane reviews. Syst Rev 2015;(1):MR000023
  • 4. Mayhew AD, Kabir M, Ansari MT. Considerations from the risk of bias perspective for updating Cochrane reviews. Syst Rev 2015;4: 136ArticlePubMedPMC
  • 5. Higgins JP, Green S, Scholten RJ. Maintaining reviews: updates, amendments and feedback. In: Higgins JP, Green S, editors. Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Chichester: Wiley-Blackwell; 2008. p. 31-46
  • 6. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007;4(3):e78ArticlePubMedPMC
  • 7. Pieper D, Antoine SL, Neugebauer EA, Eikermann M. Up-todateness of reviews is often neglected in overviews: a systematic review. J Clin Epidemiol 2014;67(12):1302-1308ArticlePubMed
  • 8. Vernooij RW, Sanabria AJ, Solà I, Alonso-Coello P, Martínez García L. Guidance for updating clinical practice guidelines: a systematic review of methodological handbooks. Implement Sci 2014;9: 3ArticlePubMedPMC
  • 9. Martínez García L, Arévalo-Rodríguez I, Solà I, Haynes RB, Vandvik PO, Alonso-Coello P, et al. Strategies for monitoring and updating clinical practice guidelines: a systematic review. Implement Sci 2012;7: 109ArticlePubMedPMC
  • 10. Crumley ET, Wiebe N, Cramer K, Klassen TP, Hartling L. Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review. BMC Med Res Methodol 2005;5: 24ArticlePubMedPMC
  • 11. Bae JM. Human papillomavirus 16 infection as a potential risk factor for prostate cancer: an adaptive meta-analysis. Epidemiol Health 2015;37: e2015005Article
  • 12. Bae JM. The necessity of an observational study on the interactions between allergic history and citrus fruit intake for the prevention of pancreatic cancer. Epidemiol Health 2015;37: e2015028Article
  • 13. Bae JM, Kim EH. Hormonal replacement therapy and the risk of lung cancer in women: an adaptive meta-analysis of cohort studies. J Prev Med Public Health 2015;48(6):280-286ArticlePubMedPMCPDF
  • 14. Bae JM, Kim EH. Human papillomavirus infection and risk of lung cancer in never-smokers and women: an ‘adaptive’ meta-analysis. Epidemiol Health 2015;37: e2015052Article
  • 15. Bae JM, Lee EJ, Guyatt G. Citrus fruit intake and pancreatic cancer risk: a quantitative systematic review. Pancreas 2009;38(2):168-174ArticlePubMed
  • 16. Pattanittum P, Laopaiboon M, Moher D, Lumbiganon P, Ngamjarus C. A comparison of statistical methods for identifying out-of-date systematic reviews. PLoS One 2012;7(11):e48894ArticlePubMedPMC
  • 17. Bae JM. Meta-epidemiology. Epidemiol Health 2014;36: e2014019Article

Figure & Data

References

    Citations

    Citations to this article as recorded by  
    • Exploring pull migration drivers to the European Union: A systematic literature review and future research agenda
      Vilmantė Kumpikaitė-Valiūnienė, Ineta Žičkutė-Daugelavičienė, Ewa Glińska, Ewa Rollnik-Sadowska, Audra I. Mockaitis
      International Journal of Intercultural Relations.2025; 109: 102296.     CrossRef
    • The impacts of menopausal hormone therapy on longer-term health consequences of ovarian hormone deficiency
      B.-K. Yoon
      Climacteric.2023; 26(3): 193.     CrossRef
    • Citation tracking for systematic literature searching: A scoping review
      Julian Hirt, Thomas Nordhausen, Christian Appenzeller‐Herzog, Hannah Ewald
      Research Synthesis Methods.2023; 14(3): 563.     CrossRef
    • Circulating 25-hydroxyvitamin D levels and hypertension risk after adjusting for publication bias
      Jong-Myon Bae
      Clinical Hypertension.2022;[Epub]     CrossRef
    • Coffee Consumption and Risk of Prostate Cancer in Local, Advanced, and Fatal Grades: A Meta-Epidemiological Study of Prospective Cohort Studies
      Jong-Myon Bae
      The Korean Journal of Urological Oncology.2021; 19(1): 16.     CrossRef
    • Sex as an effect modifier in the association between alcohol intake and gastric cancer risk
      Jong-Myon Bae
      World Journal of Gastrointestinal Oncology.2021; 13(5): 453.     CrossRef
    • Coffee consumption and risk of type 2 diabetes mellitus in Asians: A meta-epidemiological study of population-based cohort studies
      Jong-Myon Bae
      World Journal of Diabetes.2021; 12(6): 908.     CrossRef
    • Hormonal Replacement Therapy and Risk of Thyroid Cancer in Women: A Meta-Epidemiological Analysis of Prospective Cohort Studies
      Jong-Myon Bae
      Journal of Menopausal Medicine.2021; 27(3): 141.     CrossRef
    • History of Diabetes Mellitus and Risk of Breast Cancer in Asian Women: A Meta-Epidemiological Analysis of Population-Based Cohort Studies
      Jong-Myon Bae
      Journal of Menopausal Medicine.2020; 26(1): 29.     CrossRef
    • Body Mass Index and Risk of Gastric Cancer in Asian Adults: A Meta-Epidemiological Meta-Analysis of Population-Based Cohort Studies
      Jong-Myon Bae
      Cancer Research and Treatment.2020; 52(2): 369.     CrossRef
    • History of Coffee Consumption and Risk of Alzheimer's Disease: a Meta-epidemiological Study of Population-based Cohort Studies
      Jong-Myon Bae
      Dementia and Neurocognitive Disorders.2020; 19(3): 108.     CrossRef
    • History of Diabetes Mellitus and Risk of Prostate Cancer: A Meta-Epidemiological Study of Population-Based Cohort Studies in East Asian Men
      Jong-Myon Bae
      The Korean Journal of Urological Oncology.2019; 17(3): 119.     CrossRef
    • Prophylactic efficacy of probiotics on travelers’ diarrhea: an adaptive meta-analysis of randomized controlled trials
      Jong-Myon Bae
      Epidemiology and Health.2018; 40: e2018043.     CrossRef
    • The Role of Menopausal Hormone Therapy in Reducing All-cause Mortality in Postmenopausal Women Younger than 60 Years: An Adaptive Meta-analysis
      Jong-Myon Bae, Byung-Koo Yoon
      Journal of Menopausal Medicine.2018; 24(3): 139.     CrossRef
    • Is there evidence that Kudoa septempunctata can cause an outbreak of acute food poisoning?
      Young-Bae Chung, Jong-Myon Bae
      Epidemiology and Health.2017; 39: e2017004.     CrossRef
    • Human papillomavirus infection and risk of breast cancer: a meta-analysis of case-control studies
      Jong-Myon Bae, Eun Hee Kim
      Infectious Agents and Cancer.2016;[Epub]     CrossRef
    • Epstein-Barr Virus Infection and Risk of Breast Cancer: An Adaptive Meta-Analysis for Case-Control Studies
      Jong-Myon Bae, Eun Hee Kim
      Archives of Clinical Infectious Diseases.2016;[Epub]     CrossRef
    • Dietary intakes of citrus fruit and risk of gastric cancer incidence: an adaptive meta-analysis of cohort studies
      Jong-Myon Bae, Eun Hee Kim
      Epidemiology and Health.2016; 38: e2016034.     CrossRef
    • Breast Density and Risk of Breast Cancer in Asian Women: A Meta-analysis of Observational Studies
      Jong-Myon Bae, Eun Hee Kim
      Journal of Preventive Medicine and Public Health.2016; 49(6): 367.     CrossRef

    Citation Discovery Tools for Conducting Adaptive Meta-analyses to Update Systematic Reviews
    Citation Discovery Tools for Conducting Adaptive Meta-analyses to Update Systematic Reviews
    Groups Scopus ‘cited by’ (n = 573)
    [A] PM_C & SC_C Overlapping with PubMed ‘cited by’ (n = 173) 154
    [B] PM_S & SC_C Overlapping with PubMed ’similar articles’ (n = 2380) 92
    [C] SC_C Only in Scopus ‘cited by’ 327
    [D] PM_C Only in PubMed ‘cited by’ 19
    Study characteristics [A] PM_C & SC_C (n = 154) [B] PM_S & SC_C (n = 92) [C] SC_C (n = 327) [D] PM_C (n = 19)
    Laboratory studies 30 0 81 1
    Studies of diseases other than pancreatic cancer 7 1 22 1
    Methodology-focused studies 0 0 11 0
    Studies on disease statistics 4 1 14 1
    Expert opinions and reviews 16 13 126 3
    Systematic reviews 16 7 15 4
    Observational studies without information on citris fruit intake 59 64 31 7
    Clinical studies 9 0 27 0
    Articles ultimately selected 13 6 0 2
    CRU AMA
    Protocol revision Mandatory Flexible
    Starting point Study protocol Articles selected by previous SR
    Database search Mandatory If needed
    Search method Search algorithm Citation discovery tools
    Update intervals Regular Any time
    Update authors Original authors Anyone
    Study design SR SR with meta-epidemiology1
    Table 1. Distribution of lists obtained from citation discovery tools in Scopus and PubMed

    PM_C, ‘cited by’ articles provided by PubMed; SC_C, ‘cited by’ articles provided by Scopus; PM_S, ‘similar articles’ provided by PubMed.

    Table 2. Distribution of study characteristics among the 4 groups in Table 1

    PM_C, ‘cited by’ articles provided by PubMed; SC_C, ‘cited by’ articles provided by Scopus; PM_S, ‘similar articles’ provided by PubMed.

    Table 3. Differences between the Cochrane review update (CRU) and adaptive meta-analysis (AMA) for updating a systematic review (SR)

    Bae [17] describes the concept of ‘meta-epidemiology’.

    Table 1.

    Table 2.

    Table 3.

    JPMPH : Journal of Preventive Medicine and Public Health
    © Korean Society for Preventive Medicine.
    TOP