Recognizing 3D part instances from a 3D point cloud is crucial for 3D
structure and scene understanding. Several learning-based approaches use
semantic segmentation and instance center prediction as training tasks and fail
to further exploit the inherent relationship between shape semantics and part
instances. In this paper, we present a new method for 3D part instance
segmentation. Our method exploits semantic segmentation to fuse nonlocal
instance features, such as center prediction, and further enhances the fusion
scheme in a multi- and cross-level way. We also propose a semantic region
center prediction task to train and leverage the prediction results to improve
the clustering of instance points. Our method outperforms existing methods with
a large-margin improvement in the PartNet benchmark. We also demonstrate that
our feature fusion scheme can be applied to other existing methods to improve
their performance in indoor scene instance segmentation tasks.

本文提出了一种基于语义分割和实例中心预测的方法，进一步利用形状语义学和部件实例之间的内在关系，同时利用多个层次的特征融合以及语义区域中心预测任务来提高实例点的聚类性能，该方法在 PartNet 基准测试中表现优异。

多级三维零件实例分割的语义分割辅助实例特征融合

Semantic Segmentation-Assisted Instance Feature Fusion for Multi-Level  3D Part Instance Segmentation

Multimodal sentiment analysis aims to extract and integrate semantic
information collected from multiple modalities to recognize the expressed
emotions and sentiment in multimodal data. This research area's major concern
lies in developing an extraordinary fusion scheme that can extract and
integrate key information from various modalities. However, one issue that may
restrict previous work to achieve a higher level is the lack of proper modeling
for the dynamics of the competition between the independence and relevance
among modalities, which could deteriorate fusion outcomes by causing the
collapse of modality-specific feature space or introducing extra noise. To
mitigate this, we propose the Bi-Bimodal Fusion Network (BBFN), a novel
end-to-end network that performs fusion (relevance increment) and separation
(difference increment) on pairwise modality representations. The two parts are
trained simultaneously such that the combat between them is simulated. The
model takes two bimodal pairs as input due to the known information imbalance
among modalities. In addition, we leverage a gated control mechanism in the
Transformer architecture to further improve the final output. Experimental
results on three datasets (CMU-MOSI, CMU-MOSEI, and UR-FUNNY) verifies that our
model significantly outperforms the SOTA. The implementation of this work is
available at this https URL

该研究提出了一个新的端到端的网络，使用双模态配对表示执行融合和分离，并利用带有门控机制的 Transformer 架构进一步提高了最终输出，在 CMU-MOSI、CMU-MOSEI 和 UR-FUNNY 三个数据集上的实验结果验证了该模型明显优于 SOTA。

双双模态融合的相关性控制多模态情感分析

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal  Sentiment Analysis

Learning depth from spherical panoramas is becoming a popular research topic
because a panorama has a full field-of-view of the environment and provides a
relatively complete description of a scene. However, applying well-studied CNNs
for perspective images to the standard representation of spherical panoramas,
i.e., the equirectangular projection, is suboptimal, as it becomes distorted
towards the poles. Another representation is the cubemap projection, which is
distortion-free but discontinued on edges and limited in the field-of-view.
This paper introduces a new framework to fuse features from the two
projections, unidirectionally feeding the cubemap features to the
equirectangular features only at the decoding stage. Unlike the recent
bidirectional fusion approach operating at both the encoding and decoding
stages, our fusion scheme is much more efficient. Besides, we also designed a
more effective fusion module for our fusion scheme. Experiments verify the
effectiveness of our proposed fusion strategy and module, and our model
achieves state-of-the-art performance on four popular datasets. Additional
experiments show that our model also has the advantages of model complexity and
generalization capability.The code is available at
this https URL

本文提出了一种将立方体贴图特征单向馈送到全景图特征的融合框架，并设计了一个更有效的融合模块。实验证明了该融合策略和模块的有效性，该模型在四个流行数据集上实现了最先进的性能。

UniFuse: 360 度全景深度估计的单向融合

UniFuse: Unidirectional Fusion for 360$^{\circ}$ Panorama Depth  Estimation

A complex visual navigation task puts an agent in different situations which
call for a diverse range of visual perception abilities. For example, to "go to
the nearest chair", the agent might need to identify a chair in a living room
using semantics, follow along a hallway using vanishing point cues, and avoid
obstacles using depth. Therefore, utilizing the appropriate visual perception
abilities based on a situational understanding of the visual environment can
empower these navigation models in unseen visual environments. We propose to
train an agent to fuse a large set of visual representations that correspond to
diverse visual perception abilities. To fully utilize each representation, we
develop an action-level representation fusion scheme, which predicts an action
candidate from each representation and adaptively consolidate these action
candidates into the final action. Furthermore, we employ a data-driven
inter-task affinity regularization to reduce redundancies and improve
generalization. Our approach leads to a significantly improved performance in
novel environments over ImageNet-pretrained baseline and other fusion methods.

通过多种视觉感知能力的训练和数据驱动的任务亲和力正则化，提出一种行动级的视觉表征融合方案，从而在新颖的环境中提高导航模型的性能。