Meta-analysis: Methods, Strengths, and Weaknesses

The growing quantity of clinical research data has created a need to find ways to effectively provide an overview of information that addresses specific medical questions. Meta-analysis is being used ever more frequently for this purpose. Therefore, it is important to recognize both the strengths and weaknesses of this analytical methodology.

Limitations of Meta-Analyses

Buyse and colleagues discuss the use of meta-analysis as an analytical tool for combining data from randomized clinical trials. They note that, in the absence of access to individual patient data, even meta-analyses based on randomized clinical trials have limitations.

Clearly, the availability of patient data is desirable, especially when trial differences need to be explored. However, in some cases, primary data are unavailable, and not all medical questions are amenable to study through randomized trials. In the era of evidence-based medicine, if meta-analysis can help organize and evaluate the nature of the available information, it serves a useful function and should be explored without these restrictive limitations.

In his discussion of meta-analysis, Thacker[1] addresses the issue of using summary data from the literature for meta-analysis. He notes that “when criticizing meta-analysis, one must distinguish between those problems that are inherent in any literature review, and those that are specifically a problem with meta-analysis.”

One problem in using information available through literature review is the variability in what is reported. A set of guidelines for reporting results from clinical trials (the Consolidated Standards of Reporting Trials [CONSORT]) has been described.[2] These guidelines are designed to ensure that adequate information is available for evaluation of the trial by readers. The routine implementation of such guidelines would facilitate consolidation of information across studies, whether through meta-analysis or other techniques.

Guidelines for Ensuring the Quality of Meta-Analyses

As Buyse et al note, to be useful, a meta-analysis must meet quality standards. To ensure quality, certain steps are necessary, which extend beyond the actual conduct of the statistical (meta) analysis. As with randomized clinical trials, the steps taken to gather the data prior to the analysis will determine whether it is even justified to proceed with the actual mathematical calculations. These steps have been nicely defined for meta-analysis projects that are based on randomized clinical trials.[3,4] Other authors have provided similar discussions that consider a broader range of study designs.[5-7]

These steps form guidelines that can be used as a checklist not only for planning a meta-analysis but also for assessing the quality of a published report. Thus, it is worth highlighting the key components of these guidelines:

1. Meta-analysis must be looked on as a research project with a protocol. There need to be well-defined primary and secondary hypotheses. The nature of studies to be included must be described, and a comprehensive search strategy to identify all qualifying studies must be developed.

Ideally, rules for inclusion and/or exclusion of studies should be formulated in advance of the search to preclude selection bias. A record should be made of all of the studies identified, with a summary of reasons for exclusion for all of the studies not used in the analysis.

Buyse et al mention the use of quality scoring. If this is to be done, the definition of “quality” should be established prospectively, prior to the search for studies, to prevent bias.

2. The analyses should be conducted using appropriate statistical techniques, recognizing the nature of the outcome measures and the study designs. Buyse et al fully discuss some key components of the statistical methodology in their review.

3. Prior to reaching any conclusions, supplementary statistical analyses need to be performed to determine whether the data present a consistent picture. Such consistency checks are important when evaluating large multicenter clinical trials, and are even more critical for meta-analysis projects.

At a minimum, publication bias and heterogeneity, two key concerns addressed by Buyse et al, must be considered. To determine if the overall conclusions have been influenced by study selection, the meta-analysis can be repeated with one or more of the original studies excluded. For these supplemental analyses, studies may be grouped based on study design characteristics (eg, specific patient entrance criteria). Because there is no single standard for “quality,” if quality scoring is used, it may be worthwhile to repeat the analyses using different quality criteria for study selection. These additional analyses are often referred to as “sensitivity analyses.” If the study results are essentially the same for these analyses as for the primary analysis, more confidence can be placed onthe conclusions.

As has been noted by numerous authors, a central registry of studies, both published and unpublished, is the ultimate solution to the problem of publication bias. In the interim, one method of determining the potential impact of publication bias on a specific meta-analysis is to determine how many studies, and of what size, would need to be added in order to change the overall conclusions of the analysis.[8]

More on the Issue of Heterogeneity

The issue of heterogeneity is worthy of further discussion. Ultimately, the question that physicians must answer is whether the study results are applicable to their patients. In this regard, a meta-analysis that includes studies with variations in patient characteristics, due either to variations in entrance criteria or just to the patient populations accessed by the study, provides reassurance that the effect reported is not limited to a narrowly defined population.

This assumes that there is not a large degree of heterogeneity in the study outcomes, however. As noted by Buyse et al, in the presence of substantial heterogeneity in outcomes, it is less important to look at ways to deal with that heterogeneity statistically than it is to study potential sources for the differences. In fact, when large differences in outcome are observed, there is little meaning to any overall summary measure that might be calculated.

Summary

Meta-analysis should be considered as one methodology for providing an overview of multiple studies. Guidelines similar to those used to conduct a meta-analysis project based on randomized clinical trials can be applied to epidemiologic studies and even case series. The structure imposed by following a careful research plan and subsequent statistical analysis can only help clarify the nature of the available data.

The only risk is that because a “meta-analysis” has been conducted, readers may assume a greater weight of evidence than the data justify. Standardization of trial reporting would help both reviewers and readers assess the validity of any specific meta-analysis and its applicability to the medical question of interest to them. Meta-analyses undertaken by those who have an adequate understanding of the background of the research question and the methodology proposed offer real potential to assist physicians in developing evidence-based guidelines for treating their patients.

References:

1. Thacker S. Meta-analysis: A quantitative approach to research integration. JAMA 259(11):1685-1689, 1988.

2. Begg C, Cho M, Eastwood S, et al: Improving the quality of reporting of randomized controlled trials. JAMA 276(8):637-639, 1996.

3. Pogue J, Yusuf S: Overcoming the limitations of current meta-analysis of randomised controlled trials. Lancet 351:47-52, 1998.

4. Cook D, Sackett D, Spitzer W: Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam consultation on meta-analysis. J Clin Epidemiol 48(1):167-171, 1995.

5. L’Abbe K, Detsky A, O’Rourke K: Meta-analysis in clinical research. Ann Intern Med 107:224-233, 1987.

6. Jones D: Meta-analysis: Weighing the evidence. Stat Med 14:137-149, 1995.

7. Oxman A, Cook D, Guyatt G: Users’ guides to the medical literature: VI. How to use an overview. JAMA 272(17):1367-1371, 1994.

8. Moher D, Olkin I: Meta-analysis of randomized controlled trials: A concern for standards. JAMA 274(24):1962-1964, 1995.