Large language models (LLMs) have garnered significant attention and
widespread usage due to their impressive performance in various tasks. However,
they are not without their own set of challenges, including issues such as
hallucinations, factual inconsistencies, and limitations in
numerical-quantitative reasoning. Evaluating LLMs in miscellaneous reasoning
tasks remains an active area of research. Prior to the breakthrough of LLMs,
Transformers had already proven successful in the medical domain, effectively
employed for various natural language understanding (NLU) tasks. Following this
trend, LLMs have also been trained and utilized in the medical domain, raising
concerns regarding factual accuracy, adherence to safety protocols, and
inherent limitations. In this paper, we focus on evaluating the natural
language inference capabilities of popular open-source and closed-source LLMs
using clinical trial reports as the dataset. We present the performance results
of each LLM and further analyze their performance on a development set,
particularly focusing on challenging instances that involve medical
abbreviations and require numerical-quantitative reasoning. Gemini, our leading
LLM, achieved a test set F1-score of 0.748, securing the ninth position on the
task scoreboard. Our work is the first of its kind, offering a thorough
examination of the inference capabilities of LLMs within the medical domain.

我们评估了在临床试验报告数据集上使用医学领域的热门开源和闭源大型语言模型的自然语言推理能力，并分析了它们在具有医学缩写和数量 - 定量推理要求的挑战性实例上的表现。Gemini，我们的领先大型语言模型，在测试集上获得了 0.748 的 F1 分数，在任务排行榜上位列第九。我们的工作是第一种在医学领域内全面检验大型语言模型的推理能力的工作。