Retrieval-augmented language models (RALMs) improve performance by accessing
long-tail and up-to-date knowledge from external data stores, but are
challenging to build. Existing approaches require either expensive
retrieval-specific modifications to LM pre-training or use post-hoc integration
of the data store that leads to suboptimal performance. We introduce
Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning
methodology that provides a third option by retrofitting any LLM with retrieval
capabilities. Our approach operates in two distinct fine-tuning steps: (1) one
updates a pre-trained LM to better use retrieved information, while (2) the
other updates the retriever to return more relevant results, as preferred by
the LM. By fine-tuning over tasks that require both knowledge utilization and
contextual awareness, we demonstrate that each stage yields significant
performance improvements, and using both leads to additional gains. Our best
model, RA-DIT 65B, achieves state-of-the-art performance across a range of
knowledge-intensive zero- and few-shot learning benchmarks, significantly
outperforming existing in-context RALM approaches by up to +8.9% in 0-shot
setting and +1.4% in 5-shot setting on average.

Retrieval-augmented language models can be enhanced through a lightweight fine-tuning methodology called RA-DIT, which effectively integrates retrieval capabilities into any pre-trained language model, resulting in state-of-the-art performance in zero- and few-shot learning scenarios.

RA-DIT: 检索增强的双指令调整

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

In this work, we present the first corpus for German Adverse Drug Reaction
(ADR) detection in patient-generated content. The data consists of 4,169 binary
annotated documents from a German patient forum, where users talk about health
issues and get advice from medical doctors. As is common in social media data
in this domain, the class labels of the corpus are very imbalanced. This and a
high topic imbalance make it a very challenging dataset, since often, the same
symptom can have several causes and is not always related to a medication
intake. We aim to encourage further multi-lingual efforts in the domain of ADR
detection and provide preliminary experiments for binary classification using
different methods of zero- and few-shot learning based on a multi-lingual
model. When fine-tuning XLM-RoBERTa first on English patient forum data and
then on the new German data, we achieve an F1-score of 37.52 for the positive
class. We make the dataset and models publicly available for the community.

本文首次提出了一个针对德国患者提供的 Adverse Drug Reaction 侦测语料库，包含了德语患者论坛中 4169 个二进制标注文件，并呈现了这个社交媒体数据的课程和主题的不平衡性，通过使用 XLM-RoBERTa 在英语和德语患者论坛数据之前进行优化，最终实现了一个 37.52 的二元分类 F1 - 得分。

从患者角度探讨检测德语不良药物反应的跨语言方法

Cross-lingual Approaches for the Detection of Adverse Drug Reactions in German from a Patient's Perspective

Large language models, which are often trained for hundreds of thousands of
compute days, have shown remarkable capabilities for zero- and few-shot
learning. Given their computational cost, these models are difficult to
replicate without significant capital. For the few that are available through
APIs, no access is granted to the full model weights, making them difficult to
study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only
pre-trained transformers ranging from 125M to 175B parameters, which we aim to
fully and responsibly share with interested researchers. We show that OPT-175B
is comparable to GPT-3, while requiring only 1/7th the carbon footprint to
develop. We are also releasing our logbook detailing the infrastructure
challenges we faced, along with code for experimenting with all of the released
models.

我们呈现了 Open Pre-trained Transformers (OPT)，一个解码器型的预训练 transformers 套件，范围从 125M 到 175B 个参数，这些我们希望与感兴趣的研究人员完全和负责任地分享。 我们展示了 OPT-175B 与 GPT-3 相媲美，同时仅需要 1/7 的碳足迹进行开发。