Cross-modality registration between 2D images from cameras and 3D point
clouds from LiDARs is a crucial task in computer vision and robotic. Previous
methods estimate 2D-3D correspondences by matching point and pixel patterns
learned by neural networks, and use Perspective-n-Points (PnP) to estimate
rigid transformation during post-processing. However, these methods struggle to
map points and pixels to a shared latent space robustly since points and pixels
have very different characteristics with patterns learned in different manners
(MLP and CNN), and they also fail to construct supervision directly on the
transformation since the PnP is non-differentiable, which leads to unstable
registration results. To address these problems, we propose to learn a
structured cross-modality latent space to represent pixel features and 3D
features via a differentiable probabilistic PnP solver. Specifically, we design
a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D
elements using both voxels and points to learn the cross-modality latent space
with pixels. We design both the voxel and pixel branch based on CNNs to operate
convolutions on voxels/pixels represented in grids, and integrate an additional
point branch to regain the information lost during voxelization. We train our
framework end-to-end by imposing supervisions directly on the predicted pose
distribution with a probabilistic PnP solver. To explore distinctive patterns
of cross-modality features, we design a novel loss with adaptive-weighted
optimization for cross-modality feature description. The experimental results
on KITTI and nuScenes datasets show significant improvements over the
state-of-the-art methods. The code and models are available at
this https URL

通过可微分的概率 PnP 求解器学习结构化的跨模态潜空间，以表示像素特征和 3D 特征，并通过 VoxelPoint-to-Pixel 匹配设计了三元组网络，该网络使用像素和点来学习潜空间。在 KITTI 和 nuScenes 数据集上的实验结果显示了与最先进方法相比的显著改进。

图像和 LiDAR 点云的可微分配准：体素点对应

Differentiable Registration of Images and LiDAR Point Clouds with  VoxelPoint-to-Pixel Matching

Even though 3D face reconstruction has achieved impressive progress, most
orthogonal projection-based face reconstruction methods can not achieve
accurate and consistent reconstruction results when the face is very close to
the camera due to the distortion under the perspective projection. In this
paper, we propose to simultaneously reconstruct 3D face mesh in the world space
and predict 2D face landmarks on the image plane to address the problem of
perspective 3D face reconstruction. Based on the predicted 3D vertices and 2D
landmarks, the 6DoF (6 Degrees of Freedom) face pose can be easily estimated by
the PnP solver to represent perspective projection. Our approach achieves 1st
place on the leader-board of the ECCV 2022 WCPA challenge and our model is
visually robust under different identities, expressions and poses. The training
code and models are released to facilitate future research.

本文提出一种同时重建世界空间中的 3D 面网格和预测图像平面上的 2D 面标记的方法，以解决透视 3D 面重建问题，该方法基于预测的 3D 顶点和 2D 标记，可轻松用 PnP 求解器估计出 6 自由度面姿态，并在 ECCV 2022 WCPA 竞赛中获得第一名，并且在不同的身份、表情和姿势下具有视觉稳健性。

通过联合网格和标记回归进行人脸透视重建

Perspective Reconstruction of Human Faces by Joint Mesh and Landmark  Regression

Establishing correspondences from image to 3D has been a key task of 6DoF
object pose estimation for a long time. To predict pose more accurately, deeply
learned dense maps replaced sparse templates. Dense methods also improved pose
estimation in the presence of occlusion. More recently researchers have shown
improvements by learning object fragments as segmentation. In this work, we
present a discrete descriptor, which can represent the object surface densely.
By incorporating a hierarchical binary grouping, we can encode the object
surface very efficiently. Moreover, we propose a coarse to fine training
strategy, which enables fine-grained correspondence prediction. Finally, by
matching predicted codes with object surface and using a PnP solver, we
estimate the 6DoF pose. Results on the public LM-O and YCB-V datasets show
major improvement over the state of the art w.r.t. ADD(-S) metric, even
surpassing RGB-D based methods in some cases.

本文提出了一种基于离散描述符的 6DoF 物体姿态估计方法，通过学习物体表面密集编码和粗到细的训练策略，实现更精确的物体姿态估计，并在公共数据集上取得了超过现有方法的结果。

ZebraPose: 6DoF 物体姿态估计的粗细表面编码

ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose  Estimation

Conventional absolute camera pose via a Perspective-n-Point (PnP) solver
often assumes that the correspondences between 2D image pixels and 3D points
are given. When the correspondences between 2D and 3D points are not known a
priori, the task becomes the much more challenging blind PnP problem. This
paper proposes a deep CNN model which simultaneously solves for both the 6-DoF
absolute camera pose and 2D--3D correspondences. Our model comprises three
neural modules connected in sequence. First, a two-stream PointNet-inspired
network is applied directly to both the 2D image keypoints and the 3D scene
points in order to extract discriminative point-wise features harnessing both
local and contextual information. Second, a global feature matching module is
employed to estimate a matchability matrix among all 2D--3D pairs. Third, the
obtained matchability matrix is fed into a classification module to
disambiguate inlier matches. The entire network is trained end-to-end, followed
by a robust model fitting (P3P-RANSAC) at test time only to recover the 6-DoF
camera pose. Extensive tests on both real and simulated data have shown that
our method substantially outperforms existing approaches, and is capable of
processing thousands of points a second with the state-of-the-art accuracy.

本文提出了一种深度 CNN 模型，该模型能够同时解决 6 自由度绝对摄像机姿态和 2D-3D 对应关系，通过全局特征匹配模块和分类模块实现了对无先验知识的 2D-3D 对应关系的盲目 PnP 问题的有效解决。