Real-time Stereo Matching is a cornerstone algorithm for many Extended
Reality (XR) applications, such as indoor 3D understanding, video pass-through,
and mixed-reality games. Despite significant advancements in deep stereo
methods, achieving real-time depth inference with high accuracy on a low-power
device remains a major challenge. One of the major difficulties is the lack of
high-quality indoor video stereo training datasets captured by head-mounted
VR/AR glasses. To address this issue, we introduce a novel video stereo
synthetic dataset that comprises photorealistic renderings of various indoor
scenes and realistic camera motion captured by a 6-DoF moving VR/AR
head-mounted display (HMD). This facilitates the evaluation of existing
approaches and promotes further research on indoor augmented reality scenarios.
Our newly proposed dataset enables us to develop a novel framework for
continuous video-rate stereo matching.
As another contribution, our dataset enables us to proposed a new video-based
stereo matching approach tailored for XR applications, which achieves real-time
inference at an impressive 134fps on a standard desktop computer, or 30fps on a
battery-powered HMD. Our key insight is that disparity and contextual
information are highly correlated and redundant between consecutive stereo
frames. By unrolling an iterative cost aggregation in time (i.e. in the
temporal dimension), we are able to distribute and reuse the aggregated
features over time. This approach leads to a substantial reduction in
computation without sacrificing accuracy. We conducted extensive evaluations
and comparisons and demonstrated that our method achieves superior performance
compared to the current state-of-the-art, making it a strong contender for
real-time stereo matching in VR/AR applications.

实时立体匹配是许多扩展现实（XR）应用的基础算法，本研究介绍了一个包含室内场景的真实视频立体合成数据集，并提出了一种在低功耗设备上实现高准确度实时深度推测的新方法。

时域立体匹配：超过 100 帧每秒的扩展现实视频立体匹配

Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended  Reality

We explore the problem of real-time stereo matching on high-res imagery. Many
state-of-the-art (SOTA) methods struggle to process high-res imagery because of
memory constraints or speed limitations. To address this issue, we propose an
end-to-end framework that searches for correspondences incrementally over a
coarse-to-fine hierarchy. Because high-res stereo datasets are relatively rare,
we introduce a dataset with high-res stereo pairs for both training and
evaluation. Our approach achieved SOTA performance on Middlebury-v3 and
KITTI-15 while running significantly faster than its competitors. The
hierarchical design also naturally allows for anytime on-demand reports of
disparity by capping intermediate coarse results, allowing us to accurately
predict disparity for near-range structures with low latency (30ms). We
demonstrate that the performance-vs-speed trade-off afforded by on-demand
hierarchies may address sensing needs for time-critical applications such as
autonomous driving.

我们提出了一种端到端的框架，通过逐层搜索相应关系来解决高分辨率图像上的实时立体匹配问题，利用所提出的数据集进行训练和评估并在速度上超过竞争对手 ，能够在低延迟（30 毫秒）内准确预测近距离结构的视差，所提出的层次结构设计可以灵活地平衡性能与速度，用于自动驾驶等时间关键的应用。

高分辨率图像上的分层深度立体匹配

Hierarchical Deep Stereo Matching on High-resolution Images

This paper presents StereoNet, the first end-to-end deep architecture for
real-time stereo matching that runs at 60 fps on an NVidia Titan X, producing
high-quality, edge-preserved, quantization-free disparity maps. A key insight
of this paper is that the network achieves a sub-pixel matching precision than
is a magnitude higher than those of traditional stereo matching approaches.
This allows us to achieve real-time performance by using a very low resolution
cost volume that encodes all the information needed to achieve high disparity
precision. Spatial precision is achieved by employing a learned edge-aware
upsampling function. Our model uses a Siamese network to extract features from
the left and right image. A first estimate of the disparity is computed in a
very low resolution cost volume, then hierarchically the model re-introduces
high-frequency details through a learned upsampling function that uses compact
pixel-to-pixel refinement networks. Leveraging color input as a guide, this
function is capable of producing high-quality edge-aware output. We achieve
compelling results on multiple benchmarks, showing how the proposed method
offers extreme flexibility at an acceptable computational budget.

提出了 StereoNet，这是第一个端到端实时立体匹配的深度学习架构，在 NVidia Titan X 上以 60fps 运行，产生高质量，边缘保留且无量化的视差图。 该网络具有超像素匹配精度的关键洞见，比传统立体匹配方法高一个数量级，通过使用低分辨率代价体编码所需的所有信息，从而实现实时性。采用学习的边缘感知上采样函数实现空间精度，并使用 Siamese 网络从左右图像提取特征。在非常低的分辨率代价体中计算视差的初步估计，然后模型通过使用紧凑的像素到像素细化网络的学习上采样函数分层地重新引入高频细节。利用颜色输入作为指南，该函数能够产生高质量的边缘感知输出，并在多个基准测试中取得了显着的结果，演示了所提出的方法在可接受的计算预算下提供了极大的灵活性。