Recent advances in language and vision push forward the research of
captioning a single image to describing visual differences between image pairs.
Suppose there are two images, I_1 and I_2, and the task is to generate a
description W_{1,2} comparing them, existing methods directly model { I_1, I_2
} -> W_{1,2} mapping without the semantic understanding of individuals. In this
paper, we introduce a Learning-to-Compare (L2C) model, which learns to
understand the semantic structures of these two images and compare them while
learning to describe each one. We demonstrate that L2C benefits from a
comparison between explicit semantic representations and single-image captions,
and generalizes better on the new testing image pairs. It outperforms the
baseline on both automatic evaluation and human evaluation for the
Birds-to-Words dataset.

本文介绍了一种 Learning-to-Compare 模型，该模型能够理解两个图像之间的语义结构并学习描述每个图像，从而有效地进行图像比较和生成描述。使用该模型可以在 Birds-to-Words 数据集上实现比基准模型更好的性能，且同时在自动评估和人类评估中表现良好。

L2C: 描述视觉差异需要对个体进行语义理解

L2C: Describing Visual Differences Needs Semantic Understanding of  Individuals

Many real-world tasks require models to compare images along multiple
similarity conditions (e.g. similarity in color, category or shape). Existing
methods often reason about these complex similarity relationships by learning
condition-aware embeddings. While such embeddings aid models in learning
different notions of similarity, they also limit their capability to generalize
to unseen categories since they require explicit labels at test time. To
address this deficiency, we propose an approach that jointly learns
representations for the different similarity conditions and their contributions
as a latent variable without explicit supervision. Comprehensive experiments
across three datasets, Polyvore-Outfits, Maryland-Polyvore and UT-Zappos50k,
demonstrate the effectiveness of our approach: our model outperforms the
state-of-the-art methods, even those that are strongly supervised with
pre-defined similarity conditions, on fill-in-the-blank, outfit compatibility
prediction and triplet prediction tasks. Finally, we show that our model learns
different visually-relevant semantic sub-spaces that allow it to generalize
well to unseen categories.

提出一种联合学习多个相似性条件的表示及其贡献的潜变量的方法，能够有效地处理图片比较问题并兼顾泛化能力。

学习相似条件无需显式监督

Learning Similarity Conditions Without Explicit Supervision

In this paper, we introduce the task of automatically generating text to
describe the differences between two similar images. We collect a new dataset
by crowd-sourcing difference descriptions for pairs of image frames extracted
from video-surveillance footage. Annotators were asked to succinctly describe
all the differences in a short paragraph. As a result, our novel dataset
provides an opportunity to explore models that align language and vision, and
capture visual salience. The dataset may also be a useful benchmark for
coherent multi-sentence generation. We perform a firstpass visual analysis that
exposes clusters of differing pixels as a proxy for object-level differences.
We propose a model that captures visual salience by using a latent variable to
align clusters of differing pixels with output sentences. We find that, for
both single-sentence generation and as well as multi-sentence generation, the
proposed model outperforms the models that use attention alone.

本文介绍了一项任务，即自动生成文本来描述两个相似图像之间的差异。我们通过众包获取了一组新的数据集，并提出了一种模型，该模型使用隐变量来根据像素聚类将不同的像素与输出句子对齐，以捕捉视觉显著性并实现语言和视觉的对准。

学习描述一对相似图片之间的差异

Learning to Describe Differences Between Pairs of Similar Images

Understanding and extracting 3D information of objects from monocular 2D
images is a fundamental problem in computer vision. In the task of 3D object
pose estimation, recent data driven deep neural network based approaches suffer
from scarcity of real images with 3D keypoint and pose annotations. Drawing
inspiration from human cognition, where the annotators use a 3D CAD model as
structural reference to acquire ground-truth viewpoints for real images; we
propose an iterative Semantic Pose Alignment Network, called iSPA-Net. Our
approach focuses on exploiting semantic 3D structural regularity to solve the
task of fine-grained pose estimation by predicting viewpoint difference between
a given pair of images. Such image comparison based approach also alleviates
the problem of data scarcity and hence enhances scalability of the proposed
approach for novel object categories with minimal annotation. The fine-grained
object pose estimator is also aided by correspondence of learned spatial
descriptor of the input image pair. The proposed pose alignment framework
enjoys the faculty to refine its initial pose estimation in consecutive
iterations by utilizing an online rendering setup along with effectiveness of a
non-uniform bin classification of pose-difference. This enables iSPA-Net to
achieve state-of-the-art performance on various real image viewpoint estimation
datasets. Further, we demonstrate effectiveness of the approach for multiple
applications. First, we show results for active object viewpoint localization
to capture images from similar pose considering only a single image as pose
reference. Second, we demonstrate the ability of the learned semantic
correspondence to perform unsupervised part-segmentation transfer using only a
single part-annotated 3D template model per object class. To encourage
reproducible research, we have released the codes for our proposed algorithm.

iSPA-Net is an iterative Semantic Pose Alignment Network that exploits semantic 3D structural regularity to solve the task of fine-grained pose estimation by predicting viewpoint difference between a given pair of images, and achieves state-of-the-art performance on various real image viewpoint estimation datasets with the aid of correspondence of learned spatial descriptor of the input image pair and refinement in consecutive iterations utilizing an online rendering setup along with effectiveness of a non-uniform bin classification of pose-difference; the approach also shows effectiveness for active object viewpoint localization and unsupervised part-segmentation transfer using only a single part-annotated 3D template model per object class.