Models trained on semantically related datasets and tasks exhibit comparable
inter-sample relations within their latent spaces. We investigate in this study
the aggregation of such latent spaces to create a unified space encompassing
the combined information. To this end, we introduce Relative Latent Space
Aggregation, a two-step approach that first renders the spaces comparable using
relative representations, and then aggregates them via a simple mean. We
carefully divide a classification problem into a series of learning tasks under
three different settings: sharing samples, classes, or neither. We then train a
model on each task and aggregate the resulting latent spaces. We compare the
aggregated space with that derived from an end-to-end model trained over all
tasks and show that the two spaces are similar. We then observe that the
aggregated space is better suited for classification, and empirically
demonstrate that it is due to the unique imprints left by task-specific
embedders within the representations. We finally test our framework in
scenarios where no shared region exists and show that it can still be used to
merge the spaces, albeit with diminished benefits over naive merging.

通过在语义相关的数据集和任务上训练的模型，我们研究了聚合这些潜在空间以创建包含组合信息的统一空间。我们引入了相对潜在空间聚合（Relative Latent Space Aggregation）作为一个两步方法，首先使用相对表示使空间可比较，然后通过简单平均聚合它们。我们把一个分类问题细分为三种不同的设置下的学习任务，并训练每个任务的模型并聚合结果的潜在空间。我们将聚合空间与在所有任务上训练的端到端模型产生的空间进行比较，并表明两个空间是相似的。我们观察到聚合空间更适合于分类，并通过实验证明这是由于任务特定嵌入器遗留在表示中的独特印记。最后，在不存在共享区域的情况下测试我们的框架，并显示它仍然可以用于合并空间，尽管相对于简单合并而言其好处有所降低。