Large language models (LLM) learn diverse knowledge present in the
large-scale training dataset via self-supervised training. Followed by
instruction-tuning, LLM acquires the ability to return correct information for
diverse questions. However, adapting these pre-trained LLMs to new target
domains, such as different organizations or periods, for the question-answering
(QA) task incurs a substantial annotation cost. To tackle this challenge, we
propose a novel task, unsupervised LLM adaptation for question answering. In
this task, we leverage a pre-trained LLM, a publicly available QA dataset
(source data), and unlabeled documents from the target domain. Our goal is to
learn LLM that can answer questions about the target domain. We introduce one
synthetic and two real datasets to evaluate models fine-tuned on the source and
target data, and reveal intriguing insights; (i) fine-tuned models exhibit the
ability to provide correct answers for questions about the target domain even
though they do not see any questions about the information described in the
unlabeled documents, but (ii) they have difficulties in accessing information
located in the middle or at the end of documents, and (iii) this challenge can
be partially mitigated by replacing input tokens with random ones during
adaptation.

通过无监督的大语言模型适应，研究论文探索了在不同目标领域中提供正确答案的问题回答，并研究了输入标记替换等方法来部分缓解中间和末尾信息获取的困难。