Monolithic neural networks that make use of a single set of weights to learn
useful representations for downstream tasks explicitly dismiss the
compositional nature of data generation processes. This characteristic exists
in data where every instance can be regarded as the combination of an identity
concept, such as the shape of an object, combined with modifying concepts, such
as orientation, color, and size. The dismissal of compositionality is
especially detrimental in robotics, where state estimation relies heavily on
the compositional nature of physical mechanisms (e.g., rotations and
transformations) to model interactions. To accommodate this data
characteristic, modular networks have been proposed. However, a lack of
structure in each module's role, and modular network-specific issues such as
module collapse have restricted their usability. We propose a modular network
architecture that accommodates the mentioned decompositional concept by
proposing a unique structure that splits the modules into predetermined roles.
Additionally, we provide regularizations that improve the resiliency of the
modular network to the problem of module collapse while improving the
decomposition accuracy of the model.

该论文探讨神经网络的组成特性以及地位，提出了一种基于模块化的网络结构，利用预定的角色将模块拆分，同时提供对模块崩溃问题的解决方案，提高了整个网络的鲁棒性和解析准确性。

独立模块化网络

Independent Modular Networks

Vision transformers (ViTs) process input images as sequences of patches via
self-attention; a radically different architecture than convolutional neural
networks (CNNs). This makes it interesting to study the adversarial feature
space of ViT models and their transferability. In particular, we observe that
adversarial patterns found via conventional adversarial attacks show very
\emph{low} black-box transferability even for large ViT models. We show that
this phenomenon is only due to the sub-optimal attack procedures that do not
leverage the true representation potential of ViTs. A deep ViT is composed of
multiple blocks, with a consistent architecture comprising of self-attention
and feed-forward layers, where each block is capable of independently producing
a class token. Formulating an attack using only the last class token
(conventional approach) does not directly leverage the discriminative
information stored in the earlier tokens, leading to poor adversarial
transferability of ViTs. Using the compositional nature of ViT models, we
enhance transferability of existing attacks by introducing two novel strategies
specific to the architecture of ViT models. (i) Self-Ensemble: We propose a
method to find multiple discriminative pathways by dissecting a single ViT
model into an ensemble of networks. This allows explicitly utilizing
class-specific information at each ViT block. (ii) Token Refinement: We then
propose to refine the tokens to further enhance the discriminative capacity at
each block of ViT. Our token refinement systematically combines the class
tokens with structural information preserved within the patch tokens.

本研究通过提出两种攻击策略，Self-Ensemble 和 Token Refinement，充分利用了 Vision Transformers 的自注意力和组合性质来增强对抗攻击的传递性能。