We characterize the geometry and topology of the set of all weight vectors
for which a linear neural network computes the same linear transformation $W$.
This set of weight vectors is called the fiber of $W$ (under the matrix
multiplication map), and it is embedded in the Euclidean weight space of all
possible weight vectors. The fiber is an algebraic variety that is not
necessarily a manifold. We describe a natural way to stratify the fiber--that
is, to partition the algebraic variety into a finite set of manifolds of
varying dimensions called strata. We call this set of strata the rank
stratification. We derive the dimensions of these strata and the relationships
by which they adjoin each other. Although the strata are disjoint, their
closures are not. Our strata satisfy the frontier condition: if a stratum
intersects the closure of another stratum, then the former stratum is a subset
of the closure of the latter stratum. Each stratum is a manifold of class
$C^\infty$ embedded in weight space, so it has a well-defined tangent space and
normal space at every point (weight vector). We show how to determine the
subspaces tangent to and normal to a specified stratum at a specified point on
the stratum, and we construct elegant bases for those subspaces.
To help achieve these goals, we first derive what we call a Fundamental
Theorem of Linear Neural Networks, analogous to what Strang calls the
Fundamental Theorem of Linear Algebra. We show how to decompose each layer of a
linear neural network into a set of subspaces that show how information flows
through the neural network. Each stratum of the fiber represents a different
pattern by which information flows (or fails to flow) through the neural
network. The topology of a stratum depends solely on this decomposition. So
does its geometry, up to a linear transformation in weight space.

线性神经网络中的权重向量对应的纤维是一个代数多样体，称为纤维，具有层次结构的代数多样体，称为等级分层，每种模式由分层所代表，纤维的拓扑取决于权重向量的分解，以及其几何形状。

等价线性神经网络集合的几何结构

The Geometry of the Set of Equivalent Linear Neural Networks

Vision-language (VL) pre-training has recently received considerable
attention. However, most existing end-to-end pre-training approaches either
only aim to tackle VL tasks such as image-text retrieval, visual question
answering (VQA) and image captioning that test high-level understanding of
images, or only target region-level understanding for tasks such as phrase
grounding and object detection. We present FIBER (Fusion-In-the-Backbone-based
transformER), a new VL model architecture that can seamlessly handle both these
types of tasks. Instead of having dedicated transformer layers for fusion after
the uni-modal backbones, FIBER pushes multimodal fusion deep into the model by
inserting cross-attention into the image and text backbones, bringing gains in
terms of memory and performance. In addition, unlike previous work that is
either only pre-trained on image-text data or on fine-grained data with
box-level annotations, we present a two-stage pre-training strategy that uses
both these kinds of data efficiently: (i) coarse-grained pre-training based on
image-text data; followed by (ii) fine-grained pre-training based on
image-text-box data. We conduct comprehensive experiments on a wide range of VL
tasks, ranging from VQA, image captioning, and retrieval, to phrase grounding,
referring expression comprehension, and object detection. Using deep multimodal
fusion coupled with the two-stage pre-training, FIBER provides consistent
performance improvements over strong baselines across all tasks, often
outperforming methods using magnitudes more data. Code is available at
this https URL.

FIBER 是一个用于 Vision Language（VL）的新型 VL 模型结构，通过将交叉注意力插入图像和文本骨干网络，将多模态融合深入到模型中，并使用两阶段预训练策略，可以在 VL 任务中提供一致的性能提升。