When performing tasks like automatic speech recognition or spoken language
understanding for a given utterance, access to preceding text or audio provides
contextual information can improve performance. Considering the recent advances
in generative large language models (LLM), we hypothesize that an LLM could
generate useful context information using the preceding text. With appropriate
prompts, LLM could generate a prediction of the next sentence or abstractive
text like titles or topics. In this paper, we study the use of LLM-generated
context information and propose an approach to distill the generated
information during fine-tuning of self-supervised speech models, which we refer
to as generative context-aware fine-tuning. This approach allows the fine-tuned
model to make improved predictions without access to the true surrounding
segments or to the LLM at inference time, while requiring only a very small
additional context module. We evaluate the proposed approach using the SLUE and
Libri-light benchmarks for several downstream tasks: automatic speech
recognition, named entity recognition, and sentiment analysis. The results show
that generative context-aware fine-tuning outperforms a context injection
fine-tuning approach that accesses the ground-truth previous text, and is
competitive with a generative context injection fine-tuning approach that
requires the LLM at inference time.

使用生成式的上下文感知微调方法，该方法通过生成文本来提取上下文信息并改善模型预测能力，无需真实前文或语言模型在推断时的访问，且只需一个非常小的附加上下文模块，并在自监督语音模型的微调过程中进行提取和蒸馏。