A fundamental feature of human intelligence is the ability to infer
high-level abstractions from low-level sensory data. An essential component of
such inference is the ability to discover modularized generative mechanisms.
Despite many efforts to use statistical learning and pattern recognition for
finding disentangled factors, arguably human intelligence remains unmatched in
this area.
In this paper, we investigate a problem of learning, in a fully unsupervised
manner, the inverse of a set of independent mechanisms from distorted data
points. We postulate, and justify this claim with experimental results, that an
important weakness of existing machine learning solutions lies in the
insufficiency of cross-module diversification. Addressing this crucial
discrepancy between human and machine intelligence is an important challenge
for pattern recognition systems.
To this end, our work proposes an unsupervised method that discovers and
disentangles a set of independent mechanisms from unlabeled data, and learns
how to invert them. A number of experts compete against each other for
individual data points in an adversarial setting: one that best inverses the
(unknown) generative mechanism is the winner. We demonstrate that introducing
an orthogonalization layer into the expert architectures enforces additional
diversity in the outputs, leading to significantly better separability.
Moreover, we propose a procedure for relocating data points between experts to
further prevent any one from claiming multiple mechanisms. We experimentally
illustrate that these techniques allow discovery and modularization of much
less pronounced transformations, in addition to considerably faster
convergence.

本研究探究使用无监督学习的方法，从畸变的数据点中学习一组独立机制的反向模型，包括引入正交化层到专家结构中以增加输出的多样性以及通过专家之间再分配数据点的方式防止任何一个声称多种机制。

通过正交神经网络学习因果机制

Learning Causal Mechanisms through Orthogonal Neural Networks

Discussions on causal relations in real life often consider variables for
which the definition of causality is unclear since the notion of interventions
on the respective variables is obscure. Asking 'what qualifies an action for
being an intervention on the variable X' raises the question whether the action
impacted all other variables only through X or directly, which implicitly
refers to a causal model.
To avoid this known circularity, we instead suggest a notion of
'phenomenological causality' whose basic concept is a set of elementary
actions. Then the causal structure is defined such that elementary actions
change only the causal mechanism at one node (e.g. one of the causal
conditionals in the Markov factorization). This way, the Principle of
Independent Mechanisms becomes the defining property of causal structure in
domains where causality is a more abstract phenomenon rather than being an
objective fact relying on hard-wired causal links between tangible objects. We
describe this phenomenological approach to causality for toy and hypothetical
real-world examples and argue that it is consistent with the causal Markov
condition when the system under consideration interacts with other variables
that control the elementary actions.

通过基本行动的集合作为基本概念，建立元因果结构并确定有效的因果结构特性，避免了现实生活中因果关系交叉引用的循环问题，以适用于抽象领域和假想现实世界场景下的因果学习。

现象学因果关系

Phenomenological Causality

An important development in deep learning from the earliest MLPs has been a
move towards architectures with structural inductive biases which enable the
model to keep distinct sources of information and routes of processing
well-separated. This structure is linked to the notion of independent
mechanisms from the causality literature, in which a mechanism is able to
retain the same processing as irrelevant aspects of the world are changed. For
example, convnets enable separation over positions, while attention-based
architectures (especially Transformers) learn which combination of positions to
process dynamically. In this work we explore a way in which the Transformer
architecture is deficient: it represents each position with a large monolithic
hidden representation and a single set of parameters which are applied over the
entire hidden representation. This potentially throws unrelated sources of
information together, and limits the Transformer's ability to capture
independent mechanisms. To address this, we propose Transformers with
Independent Mechanisms (TIM), a new Transformer layer which divides the hidden
representation and parameters into multiple mechanisms, which only exchange
information through attention. Additionally, we propose a competition mechanism
which encourages these mechanisms to specialize over time steps, and thus be
more independent. We study TIM on a large-scale BERT model, on the Image
Transformer, and on speech enhancement and find evidence for semantically
meaningful specialization as well as improved performance.

本文提出了一种新的 Transformer 模型 ——Transformers with Independent Mechanisms (TIM)，使模型的不同部分通过注意力机制进行信息交换，从而实现对不同信息的准确分离并提高模型性能。