Large language models (LLM) learn diverse knowledge present in the
large-scale training dataset via self-supervised training. Followed by
instruction-tuning, LLM acquires the ability to return correct information for
diverse questions. However, adapting these pre-trained LLMs to new target
domains, such as different organizations or periods, for the question-answering
(QA) task incurs a substantial annotation cost. To tackle this challenge, we
propose a novel task, unsupervised LLM adaptation for question answering. In
this task, we leverage a pre-trained LLM, a publicly available QA dataset
(source data), and unlabeled documents from the target domain. Our goal is to
learn LLM that can answer questions about the target domain. We introduce one
synthetic and two real datasets to evaluate models fine-tuned on the source and
target data, and reveal intriguing insights; (i) fine-tuned models exhibit the
ability to provide correct answers for questions about the target domain even
though they do not see any questions about the information described in the
unlabeled documents, but (ii) they have difficulties in accessing information
located in the middle or at the end of documents, and (iii) this challenge can
be partially mitigated by replacing input tokens with random ones during
adaptation.

通过无监督的大语言模型适应，研究论文探索了在不同目标领域中提供正确答案的问题回答，并研究了输入标记替换等方法来部分缓解中间和末尾信息获取的困难。

无监督的问答模型自适应

Unsupervised LLM Adaptation for Question Answering

Dense retrieval models, which aim at retrieving the most relevant document
for an input query on a dense representation space, have gained considerable
attention for their remarkable success. Yet, dense models require a vast amount
of labeled training data for notable performance, whereas it is often
challenging to acquire query-document pairs annotated by humans. To tackle this
problem, we propose a simple but effective Document Augmentation for dense
Retrieval (DAR) framework, which augments the representations of documents with
their interpolation and perturbation. We validate the performance of DAR on
retrieval tasks with two benchmark datasets, showing that the proposed DAR
significantly outperforms relevant baselines on the dense retrieval of both the
labeled and unlabeled documents.

提出一种 DAR 方案，通过对文档的内插和扰动来增强文档表示，实现无监督学习的稠密检索，显著优于相关基线。

使用插值和扰动增强密集检索中的文档表示

Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation

We consider a document classification problem where document labels are
absent but only relevant keywords of a target class and unlabeled documents are
given. Although heuristic methods based on pseudo-labeling have been
considered, theoretical understanding of this problem has still been limited.
Moreover, previous methods cannot easily incorporate well-developed techniques
in supervised text classification. In this paper, we propose a theoretically
guaranteed learning framework that is simple to implement and has flexible
choices of models, e.g., linear models or neural networks. We demonstrate how
to optimize the area under the receiver operating characteristic curve (AUC)
effectively and also discuss how to adjust it to optimize other well-known
evaluation metrics such as the accuracy and F1-measure. Finally, we show the
effectiveness of our framework using benchmark datasets.

本文提出了一个理论上有保证的学习框架，用于无标签文档的分类问题，该框架可以灵活选择模型，并且有效地优化了接收器操作特征曲线下的面积，并在基准数据集上展示了其有效性。