Reading the evidence
What a meta-analysis can, and cannot, tell you.
A study of studies — the most cited form of evidence in health research, and one of the easiest to over-read. What pooling adds, what it cannot fix, and how to read one with care.
A meta-analysis is a statistical method that combines the results of several separate studies on the same question into one pooled estimate. It is usually the last step of a systematic review — a search for every relevant study, carried out by a method written down before the search begins. Done well, it tells you more than any one study can. Done carelessly, it lends the authority of a large number to a small and uneven body of work.
Meta-analysis and systematic review are not the same thing.
The two words are often used together, and they describe different work. The systematic review is the search and the appraisal: which databases were searched, which studies qualified, and how each was judged for quality. The meta-analysis is the arithmetic that may follow — weighting each study, usually by its size and precision, and pooling the results.
A systematic review can stand without a meta-analysis when the studies are too different to combine honestly. The reverse is the case to watch: a pooled number with no systematic search behind it may simply reflect which studies the authors happened to choose. The methods behind both are set out in the Cochrane Handbook for Systematic Reviews of Interventions, and the reporting standard most journals ask for is PRISMA.
One trial, or many
What a single trial can tell you.
A randomized trial answers one question, in one group of people, over one stretch of time. When it is large, carefully blinded and registered in advance, it can be very persuasive. Its limits are those of any single observation: chance, the particular population enrolled, and the choices of one research team.
What pooling adds.
Combining several trials gives a more precise estimate than any one of them, and — more usefully — shows whether independent teams in different populations found the same thing. Consistency across studies is one of the strongest signals in the literature; a result that appears only once is a result that has not yet been tested.
What a meta-analysis cannot tell you.
- It cannot repair weak studies. Pooling ten poorly controlled studies produces a precise estimate of a poorly controlled answer. A good review grades each study for risk of bias and says how much of the evidence is weak.
- It cannot see what was never published. Studies with unremarkable results are less likely to reach print. When they are missing, the pooled result leans in the direction of the studies that were.
- It can average questions that were never the same. Different doses, durations, populations and outcomes pooled together can produce a number that describes none of them. Reviewers call this heterogeneity, and a careful review measures it and explains it rather than averaging past it.
- Pooled observational studies still show association. Combining many cohort studies narrows the estimate of a correlation; it does not turn the correlation into evidence of cause.
- It is not the last word. A 1997 comparison in the New England Journal of Medicine set twelve large randomized trials against meta-analyses published earlier on the same questions, and found the earlier meta-analyses failed to predict the later trials’ results about 35 percent of the time.
How to read one
Five questions to put to any meta-analysis.
- Was the method written down first? A protocol registered in advance — on a register such as PROSPERO — makes it harder to choose the studies after seeing the answer.
- How were the studies found? Several databases, unpublished trial registers and no language limits are signs of a search meant to find everything rather than something.
- How good were the studies pooled? Look for a risk-of-bias assessment of each study, and whether the conclusion still holds when only the strongest studies are counted.
- Did the studies agree? Wide disagreement between studies is reported as heterogeneity. A single pooled number drawn from studies that point in different directions deserves less confidence.
- Who paid for the review and the trials in it? Funding is not proof of error, but it is a known source of bias — see how to evaluate a research source for how to check it.
These questions follow the considerations behind established appraisal frameworks such as GRADE and the CEBM levels of evidence. They are how we read a review before it shapes a rating — the full method is on how we rate, and the citation standard on evidence.
Common questions
- What is a meta-analysis in research?
- A meta-analysis is a statistical method that combines the results of several separate studies on the same question into a single pooled estimate. It is usually the final step of a systematic review, which first searches for every relevant study by a method stated in advance.
- What is the difference between a meta-analysis and a systematic review?
- A systematic review is the search and appraisal: a pre-stated method for finding, selecting and judging every relevant study. A meta-analysis is the arithmetic that may follow, pooling the numbers. A systematic review can exist without a meta-analysis when the studies are too different to combine; a meta-analysis without a systematic search behind it deserves more caution.
- Is a meta-analysis better than a single randomized trial?
- Not automatically. A well-conducted meta-analysis of well-conducted trials is usually the strongest evidence available on a question, because it rests on more people and more than one research team. But pooling cannot repair weak studies, and a single large, well-run trial can contradict an earlier meta-analysis. The quality of what goes in decides the value of what comes out.
- Can a meta-analysis be wrong?
- Yes. It inherits the flaws of the studies it pools, it can be skewed when unfavorable studies were never published, and it can mislead when it averages studies that asked different questions. A 1997 comparison in the New England Journal of Medicine found that earlier meta-analyses failed to predict the results of later large trials about 35 percent of the time.
- Does Temple Protocol use meta-analyses in its ratings?
- Systematic reviews and meta-analyses of randomized trials carry the most weight in our evidence standard, and they are read with the same scrutiny as any other source — how the studies were found, how good they were, how consistent, and who paid for them. The rating method is published on our how-we-rate page.
- Is this medical advice?
- No. This page explains how research is read; it says nothing about whether any treatment, supplement or practice is right for you. Decisions about your health belong with you and a qualified clinician.
This page explains how research is read. It does not say whether any treatment, supplement or practice is right for you — discuss that with your physician. See our medical disclaimer.
See how the rating works