Learning-based multi-view stereo (MVS) method heavily relies on feature
matching, which requires distinctive and descriptive representations. An
effective solution is to apply non-local feature aggregation, e.g.,
Transformer. Albeit useful, these techniques introduce heavy computation
overheads for MVS. Each pixel densely attends to the whole image. In contrast,
we propose to constrain non-local feature augmentation within a pair of lines:
each point only attends the corresponding pair of epipolar lines. Our idea
takes inspiration from the classic epipolar geometry, which shows that one
point with different depth hypotheses will be projected to the epipolar line on
the other view. This constraint reduces the 2D search space into the epipolar
line in stereo matching. Similarly, this suggests that the matching of MVS is
to distinguish a series of points lying on the same line. Inspired by this
point-to-line search, we devise a line-to-point non-local augmentation
strategy. We first devise an optimized searching algorithm to split the 2D
feature maps into epipolar line pairs. Then, an Epipolar Transformer (ET)
performs non-local feature augmentation among epipolar line pairs. We
incorporate the ET into a learning-based MVS baseline, named ET-MVSNet.
ET-MVSNet achieves state-of-the-art reconstruction performance on both the DTU
and Tanks-and-Temples benchmark with high efficiency. Code is available at
this https URL

应用非局部特征增强和基于双线搜索的 Transformer 方法改进多视角立体视觉的特征匹配，并在学习式 MVS 基线 ET-MVSNet 中加入 Epipolar Transformer 技术，提高了重建性能。

当极线约束遇上多视点立体的非局部算子

When Epipolar Constraint Meets Non-local Operators in Multi-View Stereo

Learning-based Multi-View Stereo (MVS) methods warp source images into the
reference camera frustum to form 3D volumes, which are fused as a cost volume
to be regularized by subsequent networks. The fusing step plays a vital role in
bridging 2D semantics and 3D spatial associations. However, previous methods
utilize extra networks to learn 2D information as fusing cues, underusing 3D
spatial correlations and bringing additional computation costs. Therefore, we
present MVSTER, which leverages the proposed epipolar Transformer to learn both
2D semantics and 3D spatial associations efficiently. Specifically, the
epipolar Transformer utilizes a detachable monocular depth estimator to enhance
2D semantics and uses cross-attention to construct data-dependent 3D
associations along epipolar line. Additionally, MVSTER is built in a cascade
structure, where entropy-regularized optimal transport is leveraged to
propagate finer depth estimations in each stage. Extensive experiments show
MVSTER achieves state-of-the-art reconstruction performance with significantly
higher efficiency: Compared with MVSNet and CasMVSNet, our MVSTER achieves 34%
and 14% relative improvements on the DTU benchmark, with 80% and 51% relative
reductions in running time. MVSTER also ranks first on Tanks&Temples-Advanced
among all published works. Code is released at this https URL

本文介绍一种名为 MVSTER 的新型多视角立体重建方法，该方法利用提出的极线 Transformer 以高效地学习 2D 和 3D 信息，采用级联结构以构建更精细的深度估计，通过熵正则化的最优传输算法来提高重建性能并显著减少运行时间。

MVSTER：高效的多视角立体匹配的基于极线变换机制的转换器

MVSTER: Epipolar Transformer for Efficient Multi-View Stereo

A common approach to localize 3D human joints in a synchronized and
calibrated multi-view setup consists of two-steps: (1) apply a 2D detector
separately on each view to localize joints in 2D, and (2) perform robust
triangulation on 2D detections from each view to acquire the 3D joint
locations. However, in step 1, the 2D detector is limited to solving
challenging cases which could potentially be better resolved in 3D, such as
occlusions and oblique viewing angles, purely in 2D without leveraging any 3D
information. Therefore, we propose the differentiable "epipolar transformer",
which enables the 2D detector to leverage 3D-aware features to improve 2D pose
estimation. The intuition is: given a 2D location p in the current view, we
would like to first find its corresponding point p' in a neighboring view, and
then combine the features at p' with the features at p, thus leading to a
3D-aware feature at p. Inspired by stereo matching, the epipolar transformer
leverages epipolar constraints and feature matching to approximate the features
at p'. Experiments on InterHand and Human3.6M show that our approach has
consistent improvements over the baselines. Specifically, in the condition
where no external data is used, our Human3.6M model trained with ResNet-50
backbone and image size 256 x 256 outperforms state-of-the-art by 4.23 mm and
achieves MPJPE 26.9 mm.

本文提出一种不同于传统的两步法的可微分配准方法，利用视差约束和特征匹配改进了 2D 姿态估计以提高 3D human joints 定位的精度。