Feed-forward networks are widely used in cross-modal applications to bridge
modalities by mapping distributed vectors of one modality to the other, or to a
shared space. The predicted vectors are then used to perform e.g., retrieval or
labeling. Thus, the success of the whole system relies on the ability of the
mapping to make the neighborhood structure (i.e., the pairwise similarities) of
the predicted vectors akin to that of the target vectors. However, whether this
is achieved has not been investigated yet. Here, we propose a new similarity
measure and two ad hoc experiments to shed light on this issue. In three
cross-modal benchmarks we learn a large number of language-to-vision and
vision-to-language neural network mappings (up to five layers) using a rich
diversity of image and text features and loss functions. Our results reveal
that, surprisingly, the neighborhood structure of the predicted vectors
consistently resembles more that of the input vectors than that of the target
vectors. In a second experiment, we further show that untrained nets do not
significantly disrupt the neighborhood (i.e., semantic) structure of the input
vectors.

该论文提出新的相似度衡量方法和两个实验，揭示了通过神经网络进行模态之间的映射所得到的预测向量的周围结构更像输入向量而不是目标向量，并且没有训练的网络不会显著破坏输入向量的语义结构。