Large language models (LLMs) are capable of many natural language tasks, yet
they are far from perfect. In health applications, grounding and interpreting
domain-specific and non-linguistic data is important. This paper investigates
the capacity of LLMs to deliver multi-modal health predictions based on
contextual information (e.g. user demographics, health knowledge) and
physiological data (e.g. resting heart rate, sleep minutes). We present a
comprehensive evaluation of eight state-of-the-art LLMs with diverse prompting
and fine-tuning techniques on six public health datasets (PM-Data, LifeSnaps,
GLOBEM, AW_FB, MIT-BIH & MIMIC-III). Our experiments cover thirteen consumer
health prediction tasks in mental health, activity, metabolic, sleep, and
cardiac assessment. Our fine-tuned model, Health-Alpaca exhibits comparable
performance to larger models (GPT-3.5 and GPT-4), achieving the best
performance in 5 out of 13 tasks. Ablation studies highlight the effectiveness
of context enhancement strategies, and generalization capability of the
fine-tuned models across training datasets and the size of training samples.
Notably, we observe that our context enhancement can yield up to 23.8%
improvement in performance. While constructing contextually rich prompts
(combining user context, health knowledge and temporal information) exhibits
synergistic improvement, the inclusion of health knowledge context in prompts
significantly enhances overall performance.

该研究探讨了大规模语言模型在多模态健康预测中的应用，通过上下文信息和生理数据进行综合评估，并展示了 fine-tuned 模型在健康预测任务中的表现和对上下文增强策略的有效性。