Feed-forward networks are widely used in cross-modal applications to bridge
modalities by mapping distributed vectors of one modality to the other, or to a
shared space. The predicted vectors are then used to perform e.g., retrieval or
labeling. Thus, the success of the whole system relies on the ability of the
mapping to make the neighborhood structure (i.e., the pairwise similarities) of
the predicted vectors akin to that of the target vectors. However, whether this
is achieved has not been investigated yet. Here, we propose a new similarity
measure and two ad hoc experiments to shed light on this issue. In three
cross-modal benchmarks we learn a large number of language-to-vision and
vision-to-language neural network mappings (up to five layers) using a rich
diversity of image and text features and loss functions. Our results reveal
that, surprisingly, the neighborhood structure of the predicted vectors
consistently resembles more that of the input vectors than that of the target
vectors. In a second experiment, we further show that untrained nets do not
significantly disrupt the neighborhood (i.e., semantic) structure of the input
vectors.

该论文提出新的相似度衡量方法和两个实验，揭示了通过神经网络进行模态之间的映射所得到的预测向量的周围结构更像输入向量而不是目标向量，并且没有训练的网络不会显著破坏输入向量的语义结构。

神经网络交叉模态映射真的能够桥接视觉和语言么？

Do Neural Network Cross-Modal Mappings Really Bridge Modalities?

The offset method for solving word analogies has become a standard evaluation
tool for vector-space semantic models: it is considered desirable for a space
to represent semantic relations as consistent vector offsets. We show that the
method's reliance on cosine similarity conflates offset consistency with
largely irrelevant neighborhood structure, and propose simple baselines that
should be used to improve the utility of the method in vector space evaluation.

本文研究了解决词汇类比问题的 offset 方法，揭示了该方法依赖于余弦相似度和邻域结构，提出了基于向量空间的评估方法的改进措施。

使用词语类比评估语义空间的问题

Issues in evaluating semantic spaces using word analogies

Stochastic Gradient Descent (SGD) is a workhorse in machine learning, yet its
slow convergence can be a computational bottleneck. Variance reduction
techniques such as SAG, SVRG and SAGA have been proposed to overcome this
weakness, achieving linear convergence. However, these methods are either based
on computations of full gradients at pivot points, or on keeping per data point
corrections in memory. Therefore speed-ups relative to SGD may need a minimal
number of epochs in order to materialize. This paper investigates algorithms
that can exploit neighborhood structure in the training data to share and
re-use information about past stochastic gradients across data points, which
offers advantages in the transient optimization phase. As a side-product we
provide a unified convergence analysis for a family of variance reduction
algorithms, which we call memorization algorithms. We provide experimental
results supporting our theory.

本论文研究如何利用训练数据的邻域结构来共享和复用过去随机梯度的信息，从而在瞬态优化阶段中提供优势，并提供了一个称为记忆化算法的方差减少族的统一收敛分析。