In deep learning, auxiliary objectives are often used to facilitate learning
in situations where data is scarce, or the principal task is extremely complex.
This idea is primarily inspired by the improved generalization capability
induced by solving multiple tasks simultaneously, which leads to a more robust
shared representation. Nevertheless, finding optimal auxiliary tasks that give
rise to the desired improvement is a crucial problem that often requires
hand-crafted solutions or expensive meta-learning approaches. In this paper, we
propose a novel framework, dubbed Detaux, whereby a weakly supervised
disentanglement procedure is used to discover new unrelated classification
tasks and the associated labels that can be exploited with the principal task
in any Multi-Task Learning (MTL) model. The disentanglement procedure works at
a representation level, isolating a subspace related to the principal task,
plus an arbitrary number of orthogonal subspaces. In the most disentangled
subspaces, through a clustering procedure, we generate the additional
classification tasks, and the associated labels become their representatives.
Subsequently, the original data, the labels associated with the principal task,
and the newly discovered ones can be fed into any MTL framework. Extensive
validation on both synthetic and real data, along with various ablation
studies, demonstrate promising results, revealing the potential in what has
been, so far, an unexplored connection between learning disentangled
representations and MTL. The code will be made publicly available upon
acceptance.

在深度学习中，利用辅助目标来促进学习在数据稀缺或主要任务极其复杂的情况下经常被使用。本文提出了一种名为 Detaux 的新颖框架，通过弱监督的解缠过程来发现可以与主要任务一起利用的新的无关分类任务和相关标签，从而解决了这一关键问题。在合成和真实数据上进行了广泛验证，并进行了各种消融研究，证明了解缠表示与多任务学习之间迄今未被探索的潜在联系的潜力。

解耦潜在空间促进数据驱动的辅助学习

Disentangled Latent Spaces Facilitate Data-Driven Auxiliary Learning

Auxiliary objectives, supplementary learning signals that are introduced to
help aid learning on data-starved or highly complex end-tasks, are commonplace
in machine learning. Whilst much work has been done to formulate useful
auxiliary objectives, their construction is still an art which proceeds by slow
and tedious hand-design. Intuition for how and when these objectives improve
end-task performance has also had limited theoretical backing. In this work, we
present an approach for automatically generating a suite of auxiliary
objectives. We achieve this by deconstructing existing objectives within a
novel unified taxonomy, identifying connections between them, and generating
new ones based on the uncovered structure. Next, we theoretically formalize
widely-held intuitions about how auxiliary learning improves generalization on
the end-task. This leads us to a principled and efficient algorithm for
searching the space of generated objectives to find those most useful to a
specified end-task. With natural language processing (NLP) as our domain of
study, we demonstrate that our automated auxiliary learning pipeline leads to
strong improvements over competitive baselines across continued training
experiments on a pre-trained model on 5 NLP tasks.

通过提出的自动化管道方法，我们演示了辅助学习目标如何帮助数据获取困难或高度复杂的任务，并使在自然语言处理领域的预训练模型上的持续训练实验产生了强大的改进。

AANG：自动辅助学习

AANG: Automating Auxiliary Learning

Pretrained language models have improved effectiveness on numerous tasks,
including ad-hoc retrieval. Recent work has shown that continuing to pretrain a
language model with auxiliary objectives before fine-tuning on the retrieval
task can further improve retrieval effectiveness. Unlike monolingual retrieval,
designing an appropriate auxiliary task for cross-language mappings is
challenging. To address this challenge, we use comparable Wikipedia articles in
different languages to further pretrain off-the-shelf multilingual pretrained
models before fine-tuning on the retrieval task. We show that our approach
yields improvements in retrieval effectiveness.

本文介绍了一种使用多语言维基百科文章预训练预训练语言模型的方法，以提高检索效果。

C3: 采用对比弱监督的持续预训练用于跨语言 Ad-Hoc 检索

C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval

We investigate the utility of different auxiliary objectives and training
strategies within a neural sequence labeling approach to error detection in
learner writing. Auxiliary costs provide the model with additional linguistic
information, allowing it to learn general-purpose compositional features that
can then be exploited for other objectives. Our experiments show that a joint
learning approach trained with parallel labels on in-domain data improves
performance over the previous best error detection system. While the resulting
model has the same number of parameters, the additional objectives allow it to
be optimised more efficiently and achieve better performance.

通过神经序列标记方法探究不同的辅助目标和训练策略在学习者写作错误检测中的实用性；实验结果表明，采用多目标联合学习和领域内平行标签可以提高模型性能，而不增加参数量。