Recently, multi-task instruction tuning has been applied into sentence
representation learning, which endows the capability of generating specific
representations with the guidance of task instruction, exhibiting strong
generalization ability on new tasks. However, these methods mostly neglect the
potential interference problems across different tasks and instances, which may
affect the training and convergence of the model. To address it, we propose a
data curriculum method, namely Data-CUBE, that arranges the orders of all the
multi-task data for training, to minimize the interference risks from the two
views. In the task level, we aim to find the optimal task order to minimize the
total cross-task interference risk, which is exactly the traveling salesman
problem, hence we utilize a simulated annealing algorithm to find its solution.
In the instance level, we measure the difficulty of all instances per task,
then divide them into the easy-to-difficult mini-batches for training.
Experiments on MTEB sentence representation evaluation tasks show that our
approach can boost the performance of state-of-the-art methods. Our code and
data are publicly available at the link:
https://github.com/RUCAIBox/Data-CUBE.

利用多任务指令调优，通过数据课程方法解决干扰问题，提升句子表示学习方法的性能。