Single RGB or LiDAR is the mainstream sensor for the challenging scene flow,
which relies heavily on visual features to match motion features. Compared with
single modality, existing methods adopt a fusion strategy to directly fuse the
cross-modal complementary knowledge in motion space. However, these direct
fusion methods may suffer the modality gap due to the visual intrinsic
heterogeneous nature between RGB and LiDAR, thus deteriorating motion features.
We discover that event has the homogeneous nature with RGB and LiDAR in both
visual and motion spaces. In this work, we bring the event as a bridge between
RGB and LiDAR, and propose a novel hierarchical visual-motion fusion framework
for scene flow, which explores a homogeneous space to fuse the cross-modal
complementary knowledge for physical interpretation. In visual fusion, we
discover that event has a complementarity (relative v.s. absolute) in luminance
space with RGB for high dynamic imaging, and has a complementarity (local
boundary v.s. global shape) in scene structure space with LiDAR for structure
integrity. In motion fusion, we figure out that RGB, event and LiDAR are
complementary (spatial-dense, temporal-dense v.s. spatiotemporal-sparse) to
each other in correlation space, which motivates us to fuse their motion
correlations for motion continuity. The proposed hierarchical fusion can
explicitly fuse the multimodal knowledge to progressively improve scene flow
from visual space to motion space. Extensive experiments have been performed to
verify the superiority of the proposed method.

本论文提出了一种新的分层式视觉 - 运动融合框架用于场景流，通过将事件作为 RGB 和 LiDAR 之间的桥梁，探索同质化空间来融合跨模态的互补知识，从而逐步改善场景流。

将事件融入 RGB 和 LiDAR：用于场景流的分层视觉动态融合

Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for  Scene Flow

Despite the tremendous progress of Masked Autoencoders (MAE) in developing
vision tasks such as image and video, exploring MAE in large-scale 3D point
clouds remains challenging due to the inherent irregularity. In contrast to
previous 3D MAE frameworks, which either design a complex decoder to infer
masked information from maintained regions or adopt sophisticated masking
strategies, we instead propose a much simpler paradigm. The core idea is to
apply a \textbf{G}enerative \textbf{D}ecoder for MAE (GD-MAE) to automatically
merges the surrounding context to restore the masked geometric knowledge in a
hierarchical fusion manner. In doing so, our approach is free from introducing
the heuristic design of decoders and enjoys the flexibility of exploring
various masking strategies. The corresponding part costs less than
\textbf{12\%} latency compared with conventional methods, while achieving
better performance. We demonstrate the efficacy of the proposed method on
several large-scale benchmarks: Waymo, KITTI, and ONCE. Consistent improvement
on downstream detection tasks illustrates strong robustness and generalization
capability. Not only our method reveals state-of-the-art results, but
remarkably, we achieve comparable accuracy even with \textbf{20\%} of the
labeled data on the Waymo dataset. Code will be released at
this https URL

本文提出了一种基于生成解码器的 Masked Autoencoders（GD-MAE）方法用于自动合并周围上下文信息，以恢复多层次融合遮蔽几何知识的三维点云，能在保持良好性能的情况下探索各种遮蔽策略，且实验结果表明该方法在多个大单元基准上均有一致的效果提升与强鲁棒性与泛化能力。

GD-MAE：基于 MAE 预训练的 LiDAR 点云生成解码器

GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

Multimodal sentiment analysis is a very actively growing field of research. A
promising area of opportunity in this field is to improve the multimodal fusion
mechanism. We present a novel feature fusion strategy that proceeds in a
hierarchical fashion, first fusing the modalities two in two and only then
fusing all three modalities. On multimodal sentiment analysis of individual
utterances, our strategy outperforms conventional concatenation of features by
1%, which amounts to 5% reduction in error rate. On utterance-level multimodal
sentiment analysis of multi-utterance video clips, for which current
state-of-the-art techniques incorporate contextual information from other
utterances of the same clip, our hierarchical fusion gives up to 2.4% (almost
10% error rate reduction) over currently used concatenation. The implementation
of our method is publicly available in the form of open-source code.

本论文提出了一种新的特征融合策略并应用在情感分析中，该策略通过分层融合两种模态，然后再将其与第三种模态融合，在个别话语的情感分析和视频剪辑的情感分析中均取得了优异的效果。