LLMs are revolutionizing NLP tasks. However, the most powerful LLM, like
GPT-4, is too costly for most domain-specific scenarios. We present the first
continuously trained 13B Llama2-based LLM that is purpose-built for medical
conversations and measured on automated scribing. Our results show that our
model outperforms GPT-4 in PubMedQA with 76.6\% accuracy and matches its
performance in summarizing medical conversations into SOAP notes. Notably, our
model exceeds GPT-4 in capturing a higher number of correct medical concepts
and outperforms human scribes with higher correctness and completeness.

LLMs 是 NLP 任务的革命性技术，但是像 GPT-4 这样功能强大的 LLM 对于大多数特定领域的场景来说价格过高。我们提出了第一个基于 13B Llama2 的连续训练的 LLM，该模型专门用于医学对话，并通过自动撰写进行了衡量。结果表明，我们的模型在 PubMedQA 中的准确度达到了 76.6％，并且在将医学对话总结为 SOAP 笔记方面与 GPT-4 的性能相当。值得注意的是，我们的模型能够捕捉更多正确的医学概念，超越 GPT-4，并以更高的正确性和完整性胜过人工撰稿人。

自动医疗记录生成的持续预训练语言模型方法

A Continued Pretrained LLM Approach for Automatic Medical Note  Generation

As language models are adapted by a more sophisticated and diverse set of
users, the importance of guaranteeing that they provide factually correct
information supported by verifiable sources is critical across fields of study
& professions. This is especially the case for high-stakes fields, such as
medicine and law, where the risk of propagating false information is high and
can lead to undesirable societal consequences. Previous work studying
factuality and attribution has not focused on analyzing these characteristics
of language model outputs in domain-specific scenarios. In this work, we
present an evaluation study analyzing various axes of factuality and
attribution provided in responses from a few systems, by bringing domain
experts in the loop. Specifically, we first collect expert-curated questions
from 484 participants across 32 fields of study, and then ask the same experts
to evaluate generated responses to their own questions. We also ask experts to
revise answers produced by language models, which leads to ExpertQA, a
high-quality long-form QA dataset with 2177 questions spanning 32 fields, along
with verified answers and attributions for claims in the answers.

验证与归因对于领域特定的语言模型在高风险领域中提供准确信息至关重要，本研究通过领域专家的参与，对领域特定的语言模型生成的回答进行事实性和归因方面的评估研究，并构建了包含 32 个领域，2177 个问题以及验证答案和归因的高质量长型问答数据集 ExpertQA。

ExpertQA: 专家策划的问题及答案

ExpertQA: Expert-Curated Questions and Attributed Answers

Code generation aims to generate a code snippet automatically from natural
language descriptions. Generally, the mainstream code generation methods rely
on a large amount of paired training data, including both the natural language
description and the code. However, in some domain-specific scenarios, building
such a large paired corpus for code generation is difficult because there is no
directly available pairing data, and a lot of effort is required to manually
write the code descriptions to construct a high-quality training dataset. Due
to the limited training data, the generation model cannot be well trained and
is likely to be overfitting, making the model's performance unsatisfactory for
real-world use. To this end, in this paper, we propose a task augmentation
method that incorporates domain knowledge into code generation models through
auxiliary tasks and a Subtoken-TranX model by extending the original TranX
model to support subtoken-level code generation. To verify our proposed
approach, we collect a real-world code generation dataset and conduct
experiments on it. Our experimental results demonstrate that the subtoken-level
TranX model outperforms the original TranX model and the Transformer model on
our dataset, and the exact match accuracy of Subtoken-TranX improves
significantly by 12.75% with the help of our task augmentation method. The
model performance on several code categories has satisfied the requirements for
application in industrial systems. Our proposed approach has been adopted by
Alibaba's BizCook platform. To the best of our knowledge, this is the first
domain code generation system adopted in industrial development environments.

本文提出了一种基于任务增强方法，通过辅助任务和扩展原始 TranX 模型以支持子单词级别的代码生成，将领域知识纳入代码生成模型中的方法，以解决领域特定情况下因缺乏高质量训练数据而导致代码生成模型性能不佳的问题，并验证了该方法在实际任务中的有效性。