Meta-analyses statistically aggregate the findings of different randomized
controlled trials (RCTs) to assess treatment effectiveness. Because this yields
robust estimates of treatment effectiveness, results from meta-analyses are
considered the strongest form of evidence. However, rigorous evidence syntheses
are time-consuming and labor-intensive, requiring manual extraction of data
from individual trials to be synthesized. Ideally, language technologies would
permit fully automatic meta-analysis, on demand. This requires accurately
extracting numerical results from individual trials, which has been beyond the
capabilities of natural language processing (NLP) models to date. In this work,
we evaluate whether modern large language models (LLMs) can reliably perform
this task. We annotate (and release) a modest but granular evaluation dataset
of clinical trial reports with numerical findings attached to interventions,
comparators, and outcomes. Using this dataset, we evaluate the performance of
seven LLMs applied zero-shot for the task of conditionally extracting numerical
findings from trial reports. We find that massive LLMs that can accommodate
lengthy inputs are tantalizingly close to realizing fully automatic
meta-analysis, especially for dichotomous (binary) outcomes (e.g., mortality).
However, LLMs -- including ones trained on biomedical texts -- perform poorly
when the outcome measures are complex and tallying the results requires
inference. This work charts a path toward fully automatic meta-analysis of RCTs
via LLMs, while also highlighting the limitations of existing models for this
aim.

现有的大型语言模型在处理复杂结果和推理时表现欠佳，虽然巨大的语言模型可以在二分结果方面接近完全自动的元分析。