Large language models (LLMs) exhibit in-context learning abilities which
enable the same model to perform several tasks without any task-specific
training. In contrast, traditional adaptation approaches, such as fine-tuning,
modify the underlying models for each specific task. In-context learning,
however, consistently underperforms task-specific tuning approaches even when
presented with the same examples. While most existing approaches (e.g., prompt
engineering) focus on the LLM's learned representations to patch this
performance gap, our analysis actually reveal that LLM representations contain
sufficient information to make good predictions. As such, we focus on the LLM's
reasoning abilities and demonstrate that this performance gap exists due to
their inability to perform simple probabilistic reasoning tasks. This raises an
intriguing question: Are LLMs actually capable of learning how to reason in a
task-agnostic manner? We answer this in the affirmative and propose TART which
generically improves an LLM's reasoning abilities using a synthetically trained
Transformer-based reasoning module. TART trains this reasoning module in a
task-agnostic manner using only synthetic logistic regression tasks and
composes it with an arbitrary real-world pre-trained model without any
additional training. With a single inference module, TART improves performance
across different model families (GPT-Neo, Pythia, BLOOM), model sizes (100M -
6B), tasks (14 NLP binary classification tasks), and even across different
modalities (audio and vision). Additionally, on the RAFT Benchmark, TART
improves GPT-Neo (125M)'s performance such that it outperforms BLOOM (176B),
and is within 4% of GPT-3 (175B). Our code and models are available at
this https URL .

提出了一种名为 TART 的任务无关的 Transformer-based 推理模块，使用一组合成功能，来提高不同种类、大小和模态的预训练模型（包括视觉和语音模型）在多个 NLP 分类任务上的性能，甚至使得 GPT-Neo (125M) 的性能超过 BLOOM (176B)，与 GPT-3 (175B) 的性能相差不到 4%。

TART: 适用于任务无关推理的即插即用的 Transformer 模块

TART: A plug-and-play Transformer module for task-agnostic reasoning

We study the problem of retrieval with instructions, where users of a
retrieval system explicitly describe their intent along with their queries. We
aim to develop a general-purpose task-aware retrieval system using multi-task
instruction tuning, which can follow human-written instructions to find the
best documents for a given query. We introduce the first large-scale collection
of approximately 40 retrieval datasets with instructions, BERRI, and present
TART, a multi-task retrieval system trained on BERRI with instructions. TART
shows strong capabilities to adapt to a new retrieval task via instructions and
advances the state of the art on two zero-shot retrieval benchmarks, BEIR and
LOTTE, outperforming models up to three times larger. We further introduce a
new evaluation setup, X^2-Retrieval to better reflect real-world scenarios,
where diverse domains and tasks are pooled and a system needs to find documents
aligning users' intents. In this setup, TART significantly outperforms
competitive baselines, further demonstrating the effectiveness of guiding
retrieval with instructions.

本文介绍如何使用多任务指令调整来开发一般用途基于任务感知的检索系统，并且引入了包含 40 个使用指令的检索数据集 BERRI 和在其上使用指令训练的多任务检索系统 TART。此外，本文还引入了一个新的评估环境 X ^ 2-Retrieval，其中 TART 在广泛领域和任务汇集的情况下，在用户意图对齐的情况下显著优于竞争基线。