Visual localization is of great importance in robotics and computer vision.
Recently, scene coordinate regression based methods have shown good performance
in visual localization in small static scenes. However, it still estimates
camera poses from many inferior scene coordinates. To address this problem, we
propose a novel visual localization framework that establishes 2D-to-3D
correspondences between the query image and the 3D map with a series of
learnable scene-specific landmarks. In the landmark generation stage, the 3D
surfaces of the target scene are over-segmented into mosaic patches whose
centers are regarded as the scene-specific landmarks. To robustly and
accurately recover the scene-specific landmarks, we propose the Voting with
Segmentation Network (VS-Net) to segment the pixels into different landmark
patches with a segmentation branch and estimate the landmark locations within
each patch with a landmark location voting branch. Since the number of
landmarks in a scene may reach up to 5000, training a segmentation network with
such a large number of classes is both computation and memory costly for the
commonly used cross-entropy loss. We propose a novel prototype-based triplet
loss with hard negative mining, which is able to train semantic segmentation
networks with a large number of labels efficiently. Our proposed VS-Net is
extensively tested on multiple public benchmarks and can outperform
state-of-the-art visual localization methods. Code and models are available at
\href{https://github.com/zju3dv/VS-Net}{this https URL}.

该研究针对场景坐标回归方法在视觉定位中估计相机姿态时出现的问题，提出了一种基于可学习的场景特定地标的 2D-3D 对应的视觉定位框架，并使用一种叫做 VS-Net 的网络模型解决了大量的类别不平衡问题。该方法在多个公共基准测试中表现出优异的性能表现。

VS-Net: 基于分割的投票式视觉定位

VS-Net: Voting with Segmentation for Visual Localization

The presence of a corresponding talking face has been shown to significantly
improve speech intelligibility in noisy conditions and for hearing impaired
population. In this paper, we present a system that can generate landmark
points of a talking face from an acoustic speech in real time. The system uses
a long short-term memory (LSTM) network and is trained on frontal videos of 27
different speakers with automatically extracted face landmarks. After training,
it can produce talking face landmarks from the acoustic speech of unseen
speakers and utterances. The training phase contains three key steps. We first
transform landmarks of the first video frame to pin the two eye points into two
predefined locations and apply the same transformation on all of the following
video frames. We then remove the identity information by transforming the
landmarks into a mean face shape across the entire training dataset. Finally,
we train an LSTM network that takes the first- and second-order temporal
differences of the log-mel spectrogram as input to predict face landmarks in
each frame. We evaluate our system using the mean-squared error (MSE) loss of
landmarks of lips between predicted and ground-truth landmarks as well as their
first- and second-order temporal differences. We further evaluate our system by
conducting subjective tests, where the subjects try to distinguish the real and
fake videos of talking face landmarks. Both tests show promising results.

本文介绍了一个能够从声学语音中实时生成说话人脸关键点的系统，该系统使用一个 LSTM 网络进行训练并能够在不同说话人和话语上进行产生。