Many large language models (LLMs) for medicine have largely been evaluated on
short texts, and their ability to handle longer sequences such as a complete
electronic health record (EHR) has not been systematically explored. Assessing
these models on long sequences is crucial since prior work in the general
domain has demonstrated performance degradation of LLMs on longer texts.
Motivated by this, we introduce LongBoX, a collection of seven medical datasets
in text-to-text format, designed to investigate model performance on long
sequences. Preliminary experiments reveal that both medical LLMs (e.g., BioGPT)
and strong general domain LLMs (e.g., FLAN-T5) struggle on this benchmark. We
further evaluate two techniques designed for long-sequence handling: (i)
local-global attention, and (ii) Fusion-in-Decoder (FiD). Our results
demonstrate mixed results with long-sequence handling - while scores on some
datasets increase, there is substantial room for improvement. We hope that
LongBoX facilitates the development of more effective long-sequence techniques
for the medical domain. Data and source code are available at
this https URL

评估长序列处理能力的医学大型语言模型和模型性能，引入长序列数据集 LongBoX，初步实验证明医学和一般领域大型语言模型对长序列的处理存在困难，通过两种长序列处理技术（局部 - 全局注意力、Fusion-in-Decoder）的评估，得出虽然在某些数据集上分数有所提高，但仍有较大改进空间，并希望 LongBoX 有助于开发更有效的长序列技术。