Recently, multi-task instruction tuning has been applied into sentence
representation learning, which endows the capability of generating specific
representations with the guidance of task instruction, exhibiting strong
generalization ability on new tasks. However, these methods mostly neglect the
potential interference problems across different tasks and instances, which may
affect the training and convergence of the model. To address it, we propose a
data curriculum method, namely Data-CUBE, that arranges the orders of all the
multi-task data for training, to minimize the interference risks from the two
views. In the task level, we aim to find the optimal task order to minimize the
total cross-task interference risk, which is exactly the traveling salesman
problem, hence we utilize a simulated annealing algorithm to find its solution.
In the instance level, we measure the difficulty of all instances per task,
then divide them into the easy-to-difficult mini-batches for training.
Experiments on MTEB sentence representation evaluation tasks show that our
approach can boost the performance of state-of-the-art methods. Our code and
data are publicly available at the link:
https://github.com/RUCAIBox/Data-CUBE.

利用多任务指令调优，通过数据课程方法解决干扰问题，提升句子表示学习方法的性能。

数据 CUBE: 数据课程指导的句子表征学习

Data-CUBE: Data Curriculum for Instruction-based Sentence Representation  Learning

We study the problem of retrieval with instructions, where users of a
retrieval system explicitly describe their intent along with their queries. We
aim to develop a general-purpose task-aware retrieval system using multi-task
instruction tuning, which can follow human-written instructions to find the
best documents for a given query. We introduce the first large-scale collection
of approximately 40 retrieval datasets with instructions, BERRI, and present
TART, a multi-task retrieval system trained on BERRI with instructions. TART
shows strong capabilities to adapt to a new retrieval task via instructions and
advances the state of the art on two zero-shot retrieval benchmarks, BEIR and
LOTTE, outperforming models up to three times larger. We further introduce a
new evaluation setup, X^2-Retrieval to better reflect real-world scenarios,
where diverse domains and tasks are pooled and a system needs to find documents
aligning users' intents. In this setup, TART significantly outperforms
competitive baselines, further demonstrating the effectiveness of guiding
retrieval with instructions.

本文介绍如何使用多任务指令调整来开发一般用途基于任务感知的检索系统，并且引入了包含 40 个使用指令的检索数据集 BERRI 和在其上使用指令训练的多任务检索系统 TART。此外，本文还引入了一个新的评估环境 X ^ 2-Retrieval，其中 TART 在广泛领域和任务汇集的情况下，在用户意图对齐的情况下显著优于竞争基线。