Advertisement

Systematic Reviews and Meta-Analysis of Published Studies: An Overview and Best Practices

      Abstract

      Systematic reviews and meta-analytic approaches are widely used in the clinical arena to integrate outcome data from published studies in a patient population that address a set of related research hypotheses. The credibility of this line of research is dependent on how the studies are chosen, how the data are assembled, and how the results are reported. In this brief report, we provide an overview of the minimum set of reporting requirements for systematic reviews and meta-analyses based on the Preferred Reporting Items of Systematic reviews and Meta-Analyses guidelines. As with any research, following a set of established guidelines is essential for quality and consistency of the findings across studies and for assessment of clinical utility.

      Key Words

      The Cochrane collaboration
      defines systematic review as a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. A meta-analysis then uses appropriate statistical methods to assimilate the results of the studies included in the systematic review to address a set of related research hypotheses. Meta-analysis of published studies uses summary data from the published studies and, thus, is restricted in its scope (i.e., testing new hypotheses for which summary data are not previously published). A pooled analysis on the other hand uses individual subject data from the included studies and can help answer new research hypotheses (if the relevant data are available and collected in a similar manner). Meta-analyses of published studies are typically performed when individual subject data are not readily accessible/available and under a framework when the findings from the multiple studies are conflicting or unclear. In such cases, an integration of the existing outcome data in a relatively quick manner (as opposed to a pooled analysis that requires considerably more effort) becomes important.
      As with any published research, the credibility of a systematic review is dependent on how the studies are chosen, how the data are assembled, and how the results are reported. Two recent examples of a well-conducted meta-analysis are as follows: (a) a meta-analysis performed to understand the effectiveness of low-dose computed tomography for lung cancer screening, while awaiting the results from the prospective definitive randomized trial
      • Gopal M
      • Abdullah SE
      • Grady JJ
      • et al.
      Screening for lung cancer with low-dose computed tomography: a systematic review and meta-analysis of the baseline findings of randomized controlled trials.
      and (b) a meta-analysis performed to potentially resolve the conflicting data on the choice of therapy for previously untreated, transplant ineligible, elderly patients with myeloma, while awaiting the results from the pooled analysis of individual patient data and prospective randomized trials.
      • Kapoor P
      • Rajkumar SV
      • Dispenzieri A
      • et al.
      Melphalan and prednisone versus melphalan, prednisone and thalidomide for elderly and/or transplant ineligible patients with multiple myeloma: a meta-analysis.
      Extensive work in the literature in the mid 1980s and early 1990s demonstrated that the reporting quality of systematic reviews and the approaches used for meta-analysis were generally inconsistent and incompletely reported.
      • Sacks HS
      • Berrier J
      • Reitman D
      • et al.
      Meta-analysis of randomized controlled trials.
      • Sacks HS
      • Reitman D
      • Pagano D
      • et al.
      Meta-analysis: an update.
      Thus, in 1996, the QUality of Reporting of Meta-analyses Statement was developed by a group of international researchers, which focused on the minimum set of reporting requirements for systematic reviews and meta-analyses of randomized controlled trials.
      • Moher D
      • Cook DJ
      • Eastwood S
      • for the QUOROM group
      • et al.
      Improving the quality of reporting of meta-analysis of randomized controlled trials: the QUOROM statement.
      This guideline was subsequently updated in 2009 to incorporate the conceptual and practical advances in this arena and was renamed Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA).
      • Moher D
      • Liberati A
      • Tetzlaff J
      • The PRISMA group
      • et al.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      • Liberati A
      • Altman DG
      • Tetzlaff J
      • et al.
      The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.
      The PRISMA (and previously the QUality of Reporting of Meta-analyses) guidelines have been adopted as a minimal standard for publication requirements by several journals. Although the main focus of the guidelines is on randomized trials, it can be used as a basis for reporting systematic reviews of other types of research.
      • Moher D
      • Liberati A
      • Tetzlaff J
      • The PRISMA group
      • et al.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      • Liberati A
      • Altman DG
      • Tetzlaff J
      • et al.
      The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.
      In this brief report, we provide an overview of the PRISMA guidelines, which includes the PRISMA flow diagram and the PRISMA checklist for systematic reviews and meta-analysis. We use a hypothetical example to illustrate some of the concepts. Together, the checklist and the flow diagram provide the necessary guidance and documentation essential for conducting and reporting systematic reviews and meta-analysis.

      THE PRISMA APPROACH

      The PRISMA flow diagram is a schematic representation that maps out the number of studies identified, the number excluded, and the reasons for exclusions, and the final number of studies included in the systematic review and meta-analysis. This is analogous to the CONSORT diagram that is followed when reporting results from prospective clinical trials.
      • Rennie D
      CONSORT revised—improving the reporting of randomized trials.
      The PRISMA checklist includes a comprehensive list of 27 items that pertain to the content of performing and reporting a systematic review and meta-analysis. Below is a synopsis of the main topics that is covered in the checklist.
      • Moher D
      • Liberati A
      • Tetzlaff J
      • The PRISMA group
      • et al.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      • Liberati A
      • Altman DG
      • Tetzlaff J
      • et al.
      The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.
      • a.
        Title description.
      • b.
        Abstract structure and content.
      • c.
        Introduction section that clearly outlines the rationale and the hypothesis for the purported meta-analysis.
      • d.
        Methods section, which includes the search criteria, selection process for studies that are included in the review, assembly of data, statistical approaches used to assimilate the results, and assessment of selection and/or publication bias.
      • e.
        Results section that includes a flow diagram for the final set of studies included and their quality, information on the individual study characteristics and results, bias assessment, summary of each outcome analysis with estimates and confidence intervals (CIs), and subgroup and/or sensitivity analyses results.
      • f.
        Discussion section that summarizes the main findings and highlights any limitations.
      • g.
        Role of any funding source in the systematic review to assess potential conflict of interest.
      Additional considerations in the interpretation of the findings from a meta-analysis include the weighting scheme used to adjust for the study size when integrating the results (with greater weight assigned to [larger] studies that provide more information) and the between-study heterogeneity. In a summary meta-analysis, the reasons for such heterogeneity are likely caused by the differences in (a) the definitions of the outcome across the individual studies, (b) dosage and schedule of treatments, (c) inclusion/exclusion criteria for subjects in the individual studies, etc. In the meta-analysis by Gopal et al.,
      • Gopal M
      • Abdullah SE
      • Grady JJ
      • et al.
      Screening for lung cancer with low-dose computed tomography: a systematic review and meta-analysis of the baseline findings of randomized controlled trials.
      the heterogeneity in the control arm (using chest x-ray versus no screening) across the trials was addressed by doing two separate analyses including all studies and including only those that used chest x-ray as the control arm. In the case of Kapoor et al.,
      • Kapoor P
      • Rajkumar SV
      • Dispenzieri A
      • et al.
      Melphalan and prednisone versus melphalan, prednisone and thalidomide for elderly and/or transplant ineligible patients with multiple myeloma: a meta-analysis.
      sensitivity analyses by leaving out one trial at a time were performed to more fully capture the impact of the trial level heterogeneity on the final conclusions.

      HYPOTHETICAL EXAMPLE

      Assume that six published reports testing the efficacy of a control regimen (C) to an experimental regimen (E) to treat patients with advanced lung cancer were identified using the Cochrane collaboration recommended optimal search strategy. Table 1 is a summary of the hypothetical outcome data from the individual studies. As an aside, in instances where relevant summary statistics are not readily available from the published articles, the information on observed number of events and reported p value from the log-rank-test statistic can be used to derive the point estimates and its associated standard error (or variance).
      • Parmar MK
      • Torri V
      • Stewart L
      Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints.
      Given the somewhat differing levels of agreement in the overall survival (OS) outcome for the optimal treatment (C versus E) in this example, a meta-analysis of these studies can be conducted to assimilate the results. Under a random effects model, the summary HR and the 95% CI would be 1.134 (0.89–1.44; p value: 0.3), thus suggesting that there is no statistically significant difference in OS between the treatments. Suppose that the study II outcomes were 1.5 (1.4–1.6), and 0.001 for the HR (95% CI) and the variance (log HR), and the variance (log HR) for studies 5 and 6 were 0.02, then the summary HR and the 95% CI would be 1.24 (0.99–1.6; p value: 0.06) under a random effects model, suggesting a trend toward favorable outcome for the experimental regimen E.
      TABLE 1Data from Six Hypothetical Trials with Similar Sample Size in Advanced Lung Cancer Comparing C (Control Regimen) vs. E (Experimental Regimen)
      Study NumberOverall Survival
      Hazard Ratio (C vs. E) (95% CI)Variance (Log HR)
       I1.7 (1.29–2.24)0.02
       II1.04 (0.74–1.46)0.03
       III1.5 (1.07–2.11)0.03
       IV1.3 (0.99–1.72)0.02
       V0.9 (0.78–1.03)0.005
       VI0.8 (0.75–0.85)0.001
      CI, confidence interval; HR, hazard ratio.
      The between-trial heterogeneity in both of the above scenarios is high, thus requiring additional (sensitivity/subgroup) analyses to be performed to understand and identify those studies that contribute to the heterogeneity. In this example, if studies 5 and 6 are excluded, then the heterogeneity across the remaining four trials becomes insignificant, and the overall HR of 1.4 (p = 0.002) shows a statistically significant improvement in OS for the experimental regimen over the control regimen. Removal of studies 2, 5, and 6 results in a complete disappearance of heterogeneity and a much stronger evidence in favor of E.

      SUMMARY

      The reporting quality of systematic reviews and meta-analyses has considerable variation, thus limiting the ability to comprehensively assess the strengths and weaknesses of the reviews and the clinical utility of the results. A well-conducted systematic review should be a thorough process of collecting, reviewing, and presenting all available evidence, and a well-conducted meta-analysis should use the appropriate statistical techniques to extract and combine data across studies to produce a summary result. The PRISMA guideline referenced in this brief report is a recommended standard for reporting systematic reviews and meta-analyses.

      ACKNOWLEDGMENTS

      Supported, in part, by the National Cancer Institute Grants: Mayo Clinic Cancer Center (CA-15083) and the North Central Cancer Treatment Group (CA-25224).

      REFERENCES

      1. Green S Higgins J Glossary. Cochrane Handbook for Systematic Reviews of Interventions 4.2.5. May 2005 ([updated ]. [Internet]. [cited 2010 May 20]) (Accessed May 1, 2011.)
        • Gopal M
        • Abdullah SE
        • Grady JJ
        • et al.
        Screening for lung cancer with low-dose computed tomography: a systematic review and meta-analysis of the baseline findings of randomized controlled trials.
        J Thorac Oncol. 2010; 5: 1233-1239
        • Kapoor P
        • Rajkumar SV
        • Dispenzieri A
        • et al.
        Melphalan and prednisone versus melphalan, prednisone and thalidomide for elderly and/or transplant ineligible patients with multiple myeloma: a meta-analysis.
        Leukemia. 2011; 25: 689-696
        • Sacks HS
        • Berrier J
        • Reitman D
        • et al.
        Meta-analysis of randomized controlled trials.
        New Engl J Med. 1987; 316: 450-455
        • Sacks HS
        • Reitman D
        • Pagano D
        • et al.
        Meta-analysis: an update.
        Mt Sinai J Med. 1996; 63: 216-224
        • Moher D
        • Cook DJ
        • Eastwood S
        • for the QUOROM group
        • et al.
        Improving the quality of reporting of meta-analysis of randomized controlled trials: the QUOROM statement.
        Lancet. 1999; 354: 1896-1900
        • Moher D
        • Liberati A
        • Tetzlaff J
        • The PRISMA group
        • et al.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        PLoS Med. 2009; 6: e1000097
        • Liberati A
        • Altman DG
        • Tetzlaff J
        • et al.
        The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.
        BMJ. 2009; 339: b2700
        • Rennie D
        CONSORT revised—improving the reporting of randomized trials.
        JAMA. 2001; 285: 2006-2007
        • Parmar MK
        • Torri V
        • Stewart L
        Extracting summary statistics to perform meta-analyses of the published literature for survival endpoints.
        Stat Med. 1998; 17: 2815-2834