Objective: Function is increasingly recognized as an important indicator of
whole-person health. This study evaluates the ability of publicly available
large language models (LLMs) to accurately identify the presence of functioning
information from clinical notes. We explore various strategies to improve the
performance on this task. Materials and Methods: We collect a balanced binary
classification dataset of 1000 sentences from the Mobility NER dataset, which
was curated from n2c2 clinical notes. For evaluation, we construct zero-shot
and few-shot prompts to query the LLMs whether a given sentence contains
mobility functioning information. Two sampling techniques, random sampling and
k-nearest neighbor (kNN)-based sampling, are used to select the few-shot
examples. Furthermore, we apply a parameter-efficient prompt-based fine-tuning
method to the LLMs and evaluate their performance under various training
settings. Results: Flan-T5-xxl outperforms all other models in both zero-shot
and few-shot settings, achieving a F1 score of 0.865 with a single
demonstrative example selected by kNN sampling. In prompt-based fine-tuning
experiments, this foundation model also demonstrates superior performance
across all low-resource settings, particularly achieving an impressive F1 score
of 0.922 using the full training dataset. The smaller model, Flan-T5-xl,
requires fine-tuning with only 2.3M additional parameters to achieve comparable
performance to the fully fine-tuned Gatortron-base model, both surpassing 0.9
F1 score. Conclusion: Open-source instruction-tuned LLMs demonstrate impressive
in-context learning capability in the mobility functioning classification task.
The performance of these models can be further improved by continuing
fine-tuning on a task-specific dataset.

通过对公开可用的大型语言模型进行评估，本研究探讨了识别临床记录中功能信息存在的准确性，并提出了改进性能的各种策略，结果表明 Flan-T5-xxl 模型在零样本和少样本情况下，通过 kNN 采样选择的单个举例，可达到 0.865 的 F1 得分。