Forming and using symbol-like structured representations for reasoning has
been considered essential for generalising over novel inputs. The primary tool
that allows generalisation outside training data distribution is the ability to
abstract away irrelevant information into a compact form relevant to the task.
An extreme form of such abstract representations is symbols. Humans make use of
symbols to bind information while abstracting away irrelevant parts to utilise
the information consistently and meaningfully. This work estimates the state of
such structured representations in vision encoders. Specifically, we evaluate
image encoders in large vision-language pre-trained models to address the
question of which desirable properties their representations lack by applying
the criteria of symbolic structured reasoning described for LLMs to the image
models. We test the representation space of image encoders like VIT, BLIP,
CLIP, and FLAVA to characterise the distribution of the object representations
in these models. In particular, we create decoding tasks using multi-object
scenes from the COCO dataset, relating the token space to its input content for
various objects in the scene. We use these tasks to characterise the network's
token and layer-wise information modelling. Our analysis highlights that the
CLS token, used for the downstream task, only focuses on a few objects
necessary for the trained downstream task. Still, other individual objects are
well-modelled separately by the tokens in the network originating from those
objects. We further observed a widespread distribution of scene information.
This demonstrates that information is far more entangled in tokens than optimal
for representing objects similar to symbols. Given these symbolic properties,
we show the network dynamics that cause failure modes of these models on basic
downstream tasks in a multi-object scene.

利用图像编码器评估大型视觉语言预训练模型的结构化表示状态，揭示了这些模型在基本多对象场景下的失败模式。

基于 Transformer 的视觉编码器中的表示结构有多强化？对视觉语言模型中多物体表示的分析

How structured are the representations in transformer-based vision  encoders? An analysis of multi-object representations in vision-language  models

To tackle increasingly complex tasks, it has become an essential ability of
neural networks to learn abstract representations. These task-specific
representations and, particularly, the invariances they capture turn neural
networks into black box models that lack interpretability. To open such a black
box, it is, therefore, crucial to uncover the different semantic concepts a
model has learned as well as those that it has learned to be invariant to. We
present an approach based on INNs that (i) recovers the task-specific, learned
invariances by disentangling the remaining factor of variation in the data and
that (ii) invertibly transforms these recovered invariances combined with the
model representation into an equally expressive one with accessible semantic
concepts. As a consequence, neural network representations become
understandable by providing the means to (i) expose their semantic meaning,
(ii) semantically modify a representation, and (iii) visualize individual
learned semantic concepts and invariances. Our invertible approach
significantly extends the abilities to understand black box models by enabling
post-hoc interpretations of state-of-the-art networks without compromising
their performance. Our implementation is available at
this https URL .

为了了解神经网络不透明的黑盒表示法，本研究提出了一种基于 INNs 的方法来恢复任务特定和已学习不变量，将其变成易于理解的语义概念，并在不影响性能的情况下进行后期解释。

理解 CNN：利用 INN 解释深度表示及不变性

Making Sense of CNNs: Interpreting Deep Representations & Their  Invariances with INNs

Learning object-centric representations of complex scenes is a promising step
towards enabling efficient abstract reasoning from low-level perceptual
features. Yet, most deep learning approaches learn distributed representations
that do not capture the compositional properties of natural scenes. In this
paper, we present the Slot Attention module, an architectural component that
interfaces with perceptual representations such as the output of a
convolutional neural network and produces a set of task-dependent abstract
representations which we call slots. These slots are exchangeable and can bind
to any object in the input by specializing through a competitive procedure over
multiple rounds of attention. We empirically demonstrate that Slot Attention
can extract object-centric representations that enable generalization to unseen
compositions when trained on unsupervised object discovery and supervised
property prediction tasks.

本文提出了一个名为 Slot Attention 的架构组件，它能够从低级感知特征中提取物体为中心的表示，并能够推广到未见组合。