AHRQ Series Part II: Methods Guide for Comparative Effectiveness - Guest Editor, Mark HelfandConducting quantitative synthesis when comparing medical interventions: AHRQ and the Effective Health Care Program
Introduction
Comparative effectiveness reviews (CERs) are systematic reviews that summarize comparative effectiveness and harms of alternative clinical options, and aim to help clinicians, policy makers, and patients make informed treatment choices. Quantitative synthesis, or meta-analysis, is often essential for CERs to provide scientifically rigorous summary information. Quantitative synthesis should be conducted in a transparent and consistent way, and methodologies reported explicitly. Reasons for this were made clear during the controversy around the safety of rosiglitazone, where a systematic review that found increased risk for myocardial infarction [1] spurred heated debate on issues around choosing appropriate methods for quantitative syntheses [2], [3], [4]; and the subsequent Congressional hearing [5] brought these issues further into spotlight. This story highlighted the fact that basic issues in quantitative syntheses, such as choice of an effect measure or a model, or how to handle heterogeneity, remain crucial considerations and are often the subject of controversy and debate.
A CER typically evaluates the evidence on multiple alternative interventions, whereas most published meta-analyses compared one intervention with a placebo. Inclusion of multiple interventions increases the complexity of quantitative synthesis and entails methods of comparing multiple interventions simultaneously. Evaluation of multiple interventions also makes the assessment of similarity among studies and the decision to combine studies even more challenging. Presenting results of a meta-analysis from a CER in a way that is useful to decision makers is also a challenge.
The Evidence-based Practice Center (EPC) program of the Agency for Healthcare Research and Quality (AHRQ) [6] is the leading U.S. program providing unbiased and independent CERs. The goal of this article is to summarize our recommendations in conducting quantitative synthesis of CERs for therapeutic benefits and harms for the EPC program with the goal to improve consistency and transparency. The recommendations cover recurrent issues in the EPC program and we focus on methods for combining study-level effect measures. First, we discuss considerations for deciding whether to combine studies, followed by discussions on indirect comparison and incorporation of indirect evidence. Then, we describe our recommendations for choosing effect measures and statistical models, giving special attention to combining studies with rare events; and on testing and exploring heterogeneity. Finally, we briefly present recommendations on combining studies of mixed design and on sensitivity analysis. This article is not a comprehensive review of methods.
The recommendations were developed using group discussion and consensus based on current knowledge in the literature [7]. EPC investigators are encouraged to follow these recommendations but may choose to use alternative methods if deemed appropriate. If alternative methods are used, the investigators are required to provide rationales for their choice, and if appropriate, to state the strengths and limitations of the chosen method to promote consistency and transparency. In addition, several steps in conducting a meta-analysis require subjective decisions, for example, the decision to combine studies or the decision to incorporate indirect evidence. For each subjective decision, investigators should fully explain how the decision was reached.
Section snippets
Decision to combine studies
The decision to combine studies to produce an overall estimate should depend on whether a meaningful answer to a well-formulated research question can be obtained. The purpose of a meta-analysis should be explicitly stated in the methods section of the CER. The overall purpose of the review is not in itself a justification for conducting a meta-analysis, nor is the existence of a group of studies that address the same treatments. Investigators should avoid statements such as “We conducted a
Indirect comparisons and consideration of indirect evidence
Multiple alternative interventions for a given condition usually constitute a network of treatments. In its simplest form, a network consists of three interventions, for example, interventions A, B, and C. Randomized controlled trials (RCT) of A vs. B provide direct evidence on the comparative effectiveness of A vs. B; trials of A vs. C and B vs. C would provide indirect estimates of A vs. B through the “common reference,” C. The inclusion of more interventions would form more complex networks
Choice of effect measures
Effect measures quantify differences in outcomes, either effectiveness or harms, between treatments in trials (or exposure groups in observational studies). The choice of effect measures is first determined by the type of outcomes. For example, relative risk (RR) and odds ratio (OR) are used for a binary outcome and mean difference is for a continuous outcome. They could also be broadly classified into absolute measures—such as risk differences (RDs) or mean differences—and relative
Choice of statistical model for combining studies
Meta-analysis can be performed using either a fixed or a random effects model. A fixed effects model assumes that there is one single treatment effect across studies. Generally, a fixed effects model is not advised in the presence of significant heterogeneity. In practice, clinical and methodological diversity are always present across a set of included studies. Variation among studies is inevitable whether or not the test of heterogeneity detects it. Therefore, we recommend random effects
Test and explore statistical heterogeneity
Investigators should assess the heterogeneity for each meta-analysis. Visual inspection of forest plots and cumulative meta-analysis plots [46] are useful in the initial assessment of statistical heterogeneity. A test for the presence of statistical heterogeneity, for example, Cochran’s Q test, and a measure for magnitude of heterogeneity, for example, the I2 statistic [11], [47], are useful and should be reported. Further, interpretation of Q statistic should consider the limitations of the
Combining studies of mixed designs
In principle, studies from different randomized trial designs, for example, parallel, cross-over, factorial, or cluster-randomized design, may be combined in a single meta-analysis. Investigators should perform a comprehensive evaluation of clinical and methodological diversity and statistical heterogeneity to determine whether the trials should actually be combined, and consider any important differences between different types of trials. For cross-over trials, investigators should first
Sensitivity analyses
Completing a CER is a structured process. Investigators make decisions and assumptions in the process of conducting the review and meta-analysis; each of these decisions and assumptions may affect the main findings. Sensitivity analysis should always be conducted in a meta-analysis to investigate the robustness of the results in relation to these decisions and assumptions [60]. Results are robust if decisions and assumptions only lead to small changes in the estimates and do not affect the
Concluding remarks
In this article, we provided our recommendations on important issues in meta-analyses to improve transparency and consistency in conducting CERs. The key points and recommendations for each covered issue are summarized in Table 2. Compared with the Cochrane Handbook, which explains meta-analysis methods in more detail, we focused on selected issues that present particular challenges in CERs. Overall, there is no fundamental inconsistency between our recommendations and Cochrane Handbook on
Acknowledgments
The authors would like to acknowledge Susan Norris for participating in the workgroup calls and commenting on an earlier version of this article, Ben Vandermeer for participating workgroup calls, Christopher Schmid for reviewing and commenting on the article, Mark Helfand and Edwin Reid for editing the article, and Brian Garvey for working on references and formatting the article.
Disclaimer: The views expressed in this article are those of the authors and do not represent the official policies
References (60)
- et al.
Principles for developing guidance: AHRQ and the effective health care program
J Clin Epidemiol
(2010) - et al.
The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials
J Clin Epidemiol
(1997) - et al.
Indirect comparison in evaluating relative efficacy illustrated by antimicrobial prophylaxis in colorectal surgery
Control Clin Trials
(2000) - et al.
Initial highly-active antiretroviral therapy with a protease inhibitor versus a non-nucleoside reverse transcriptase inhibitor: discrepancies between direct and indirect meta-analyses
Lancet
(2006) Indirect comparisons: the mesh and mess of clinical trials
Lancet
(2006)- et al.
Meta-analysis in clinical trials
Control Clin Trials
(1986) - et al.
Random-effects model for meta-analysis of clinical trials: an update
Contemp Clin Trials
(2007) - et al.
The binomial distribution of meta-analysis was preferred to model within-study variability
J Clin Epidemiol
(2008) - et al.
Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care
J Clin Epidemiol
(1995) - et al.
Summing up evidence: one answer is not always enough
Lancet
(1998)
Meta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors
J Clin Epidemiol
Comparing medical interventions: AHRQ and the effective health-care program
J Clin Epidemiol
Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes
New Engl J Med
Meta-analysis of rare events: an update and sensitivity analysis of cardiovascular events in randomized trials of rosiglitazone
Clin Trials
Uncertain effects of rosiglitazone on the risk for myocardial infarction and cardiovascular death
Ann Intern Med
Fixed vs random effects meta-analysis in rare event studies: the rosiglitazone link with myocardial infarction and cardiac death
Stat Med
Hearing on FDA’s role in evaluating safety of avandia
Evidence-based Practice Centers
Cochrane handbook for systematic reviews of interventions
Quantifying heterogeneity in a meta-analysis
Stat Med
Detecting and describing heterogeneity in meta-analysis
Stat Med
Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses
Stat Med
Combination of direct and indirect evidence in mixed treatment comparisons
Stat Med
Network meta-analysis for indirect treatment comparisons
Stat Med
The transitive fallacy for randomized trials: if A bests B and B bests C in separate trials, is A better than C?
BMC Med Res Methodol
Simultaneous comparison of multiple treatments: combining direct and indirect evidence
BMJ
Indirect comparisons of competing interventions
Health Technol Assess
Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses
BMJ
Meta-analysis of migraine headache treatments: combining information from heterogeneous designs
J Am Stat Assoc
Assessing evidence inconsistency in mixed treatment comparisons
J Am Stat Assoc
Cited by (423)
Meta-analysis of the implied distribution of callous-unemotional traits across sampling methods and informant
2024, Clinical Psychology ReviewPrecipitation change affects forest soil carbon inputs and pools: A global meta-analysis
2024, Science of the Total Environment
This article was written with support from the Effective Health Care Program at the U.S. Agency for Healthcare Research and Quality.