Robust meta-analytic conclusions mandate the provision of prediction intervals in meta-analysis summaries

Objectives: Results of meta-analyses typically conclude that future large studies may be mandated. However, the predictive ability of these estimates is deficient. We explored meta-analytic prediction intervals as means for providing a clear and appropriate future treatment summary reflecting current estimates. Study Design: A meta-epidemiological study of binary outcome critical care meta-analyses published between 2002 and 2010. Computation of 95% DerSimonian-Laird and Bayesian random-effects meta-analytic confidence intervals (CI) and 95% credible intervals (CrI), respectively, and frequentist (PI) and Bayesian (PrI) prediction intervals for odds ratio (OR) and risk ratio (RR) were undertaken. Bayesian calculations included the probability that the OR and RR point estimates ≥1. Results: Seventy-two meta-analyses from 70 articles were identified, containing between three and 80 studies each, with median nine studies. For both frequentist and Bayesian settings, 49-69% of the meta-analyses excluded the null. All significant CrI had high probabilities of efficacy/harm. The number of PI vs. PrI excluding 1 was 25% vs. 3% (OR), 26% vs. 3% (RR) of the total meta-analyses. Unsurprisingly, PI/PrI width was greater than CI/CrI width and increased with increasing heterogeneity and combination of fewer studies. Conclusion: Robust meta-analytic conclusions and determination of studies warranting new large trials may be more appropriately signaled by consideration of initial interval estimates with prediction intervals. Substantial heterogeneity results in exceedingly wide PIs. More caution should be exercised regarding the conclusions of a meta-analysis.