Advances in Large Language Models (LLMs) have led to significant interest in
their potential to support human experts across a range of domains, including
public health. In this work we present automated evaluations of LLMs for public
health tasks involving the classification and extraction of free text. We
combine six externally annotated datasets with seven new internally annotated
datasets to evaluate LLMs for processing text related to: health burden,
epidemiological risk factors, and public health interventions. We initially
evaluate five open-weight LLMs (7-70 billion parameters) across all tasks using
zero-shot in-context learning. We find that Llama-3-70B-Instruct is the highest
performing model, achieving the best results on 15/17 tasks (using micro-F1
scores). We see significant variation across tasks with all open-weight LLMs
scoring below 60% micro-F1 on some challenging tasks, such as Contact
Classification, while all LLMs achieve greater than 80% micro-F1 on others,
such as GI Illness Classification. For a subset of 12 tasks, we also evaluate
GPT-4 and find comparable results to Llama-3-70B-Instruct, which scores equally
or outperforms GPT-4 on 6 of the 12 tasks. Overall, based on these initial
results we find promising signs that LLMs may be useful tools for public health
experts to extract information from a wide variety of free text sources, and
support public health surveillance, research, and interventions.

基于对 LLMs 在公共卫生任务中的自动评估，将六个外部注释数据集与七个新的内部注释数据集相结合，评估 LLMs 在处理与健康负担、流行病学危险因素和公共卫生干预相关的文本中的性能，发现 Llama-3-70B-Instruct 是性能最好的模型，在 15 个任务中取得最佳结果。这些初步结果表明，LLMs 可能成为公共卫生专家从各种免费文本来源中提取信息，并支持公共卫生监测、研究和干预的有用工具。

评估大型语言模型用于公共卫生分类和提取任务

Evaluating Large Language Models for Public Health Classification and  Extraction Tasks

This paper describes our approach to the EmotionX-2019, the shared task of
SocialNLP 2019. To detect emotion for each utterance of two datasets from the
TV show Friends and Facebook chat log EmotionPush, we propose two-step deep
learning based methodology: (i) encode each of the utterance into a sequence of
vectors that represent its meaning; and (ii) use a simply softmax classifier to
predict one of the emotions amongst four candidates that an utterance may
carry. Notice that the source of labeled utterances is not rich, we utilise a
well-trained model, known as BERT, to transfer part of the knowledge learned
from a large amount of corpus to our model. We then focus on fine-tuning our
model until it well fits to the in-domain data. The performance of the proposed
model is evaluated by micro-F1 scores, i.e., 79.1% and 86.2% for the testsets
of Friends and EmotionPush, respectively. Our model ranks 3rd among 11
submissions.

本文介绍了我们通过深度学习和使用 BERT 模型的方法，在 EmotionX-2019 和 SocialNLP 2019 的共享任务中，对来自电视节目 Friends 和 Facebook 聊天记录 EmotionPush 的每个话语的情感进行了识别，并通过微平均 F1 分数对模型性能进行了评估，得到了同类中的第三名。