Multimodal representation learning techniques typically rely on paired
samples to learn common representations, but paired samples are challenging to
collect in fields such as biology where measurement devices often destroy the
samples. This paper presents an approach to address the challenge of aligning
unpaired samples across disparate modalities in multimodal representation
learning. We draw an analogy between potential outcomes in causal inference and
potential views in multimodal observations, which allows us to use Rubin's
framework to estimate a common space in which to match samples. Our approach
assumes we collect samples that are experimentally perturbed by treatments, and
uses this to estimate a propensity score from each modality, which encapsulates
all shared information between a latent state and treatment and can be used to
define a distance between samples. We experiment with two alignment techniques
that leverage this distance -- shared nearest neighbours (SNN) and optimal
transport (OT) matching -- and find that OT matching results in significant
improvements over state-of-the-art alignment approaches in both a synthetic
multi-modal setting and in real-world data from NeurIPS Multimodal Single-Cell
Integration Challenge.

通过借鉴因果推断中的潜在结果与多模态观察中的潜在视角的类比，本研究提出了一种方法来解决多模态表示学习中异质模态样本的对齐问题。该方法假设我们通过实验处理来收集样本，并使用此来估计每个模态的倾向得分，从而定义了样本之间的距离。实验结果表明，使用最优传输匹配技术相较于现有先进对齐方法在综合多模态设置和 NeurIPS 多模态单细胞整合挑战中的真实数据中能够显著提高对齐效果。

无配对多模态数据的倾向得分匹配

Propensity Score Alignment of Unpaired Multimodal Data

Monocular 3D object detection is an essential component in autonomous driving
while challenging to solve, especially for those occluded samples which are
only partially visible. Most detectors consider each 3D object as an
independent training target, inevitably resulting in a lack of useful
information for occluded samples. To this end, we propose a novel method to
improve the monocular 3D object detection by considering the relationship of
paired samples. This allows us to encode spatial constraints for
partially-occluded objects from their adjacent neighbors. Specifically, the
proposed detector computes uncertainty-aware predictions for object locations
and 3D distances for the adjacent object pairs, which are subsequently jointly
optimized by nonlinear least squares. Finally, the one-stage uncertainty-aware
prediction structure and the post-optimization module are dedicatedly
integrated for ensuring the run-time efficiency. Experiments demonstrate that
our method yields the best performance on KITTI 3D detection benchmark, by
outperforming state-of-the-art competitors by wide margins, especially for the
hard samples.

本文提出了一种考虑成对样本关系来改进单目三维物体检测的方法，包括计算对象对之间的不确定性感知预测和利用非线性最小二乘方法对其进行优化，实验结果表明，该方法在 KITTI 三维检测基准测试中表现最佳。