This paper proposes a general interpretable predictive system with shared
information. The system is able to perform predictions in a multi-task setting
where distinct tasks are not bound to have the same input/output structure.
Embeddings of input and output variables in a common space are obtained, where
the input embeddings are produced through attending to a set of shared
embeddings, reused across tasks. All the embeddings are treated as model
parameters and learned. Specific restrictions on the space of shared embedings
and the sparsity of the attention mechanism are considered. Experiments show
that the introduction of shared embeddings does not deteriorate the results
obtained from a vanilla variable embeddings method. We run a number of further
ablations. Inducing sparsity in the attention mechanism leads to both an
increase in accuracy and a significant decrease in the number of training steps
required. Shared embeddings provide a measure of interpretability in terms of
both a qualitative assessment and the ability to map specific shared embeddings
to pre-defined concepts that are not tailored to the considered model. There
seems to be a trade-off between accuracy and interpretability. The basic shared
embeddings method favors interpretability, whereas the sparse attention method
promotes accuracy. The results lead to the conclusion that variable embedding
methods may be extended with shared information to provide increased
interpretability and accuracy.

该论文提出了一种具有共享信息的通用可解释性预测系统，该系统能够在不同的多任务设置中进行预测，其中各个任务不一定具有相同的输入 / 输出结构。通过在一个共同的空间中获取输入和输出变量的嵌入，通过参考一组共享的嵌入来生成输入嵌入，这些嵌入在任务之间被重复使用。所有的嵌入都被视为模型参数并进行学习。论文通过对共享嵌入空间和注意机制的稀疏性进行具体的限制。实验证明，引入共享嵌入并不会损害从基本变量嵌入方法中得到的结果。进一步进行了一系列的剔除分析。在注意机制中引入稀疏性既提高了准确性，又显著减少了所需的训练步骤的数量。共享嵌入在定性评估和将特定共享嵌入映射到预定义概念方面提供了可解释性的度量，而这些概念不是针对所考虑的模型设计的。准确性和可解释性之间存在一种权衡。基本的共享嵌入方法注重可解释性，而稀疏注意方法则提倡准确性。研究结果表明，可将变量嵌入方法扩展为共享信息方法，以提供更高的可解释性和准确性。

可解释的多任务学习与共享的变量嵌入

Interpretable Multi-task Learning with Shared Variable Embeddings

Intelligent behaviour in the real-world requires the ability to acquire new
knowledge from an ongoing sequence of experiences while preserving and reusing
past knowledge. We propose a novel algorithm for unsupervised representation
learning from piece-wise stationary visual data: Variational Autoencoder with
Shared Embeddings (VASE). Based on the Minimum Description Length principle,
VASE automatically detects shifts in the data distribution and allocates spare
representational capacity to new knowledge, while simultaneously protecting
previously learnt representations from catastrophic forgetting. Our approach
encourages the learnt representations to be disentangled, which imparts a
number of desirable properties: VASE can deal sensibly with ambiguous inputs,
it can enhance its own representations through imagination-based exploration,
and most importantly, it exhibits semantically meaningful sharing of latents
between different datasets. Compared to baselines with entangled
representations, our approach is able to reason beyond surface-level statistics
and perform semantically meaningful cross-domain inference.

本文提出了一种基于最小描述长度原则的变分自编码器算法，可以实现无监督表示学习，并可以自动检测数据分布的变化并为新知识分配足够的表示空间，同时保护以前学习的表示免于灾难性遗忘，该方法可以实现表示的分离，具有共享潜在特征的语义上有意义的特征，同时可以跨领域共享经验。