Semantic scene completion (SSC) aims to predict complete 3D voxel occupancy
and semantics from a single-view RGB-D image, and recent SSC methods commonly
adopt multi-modal inputs. However, our investigation reveals two limitations:
ineffective feature learning from single modalities and overfitting to limited
datasets. To address these issues, this paper proposes a novel SSC framework -
Adversarial Modality Modulation Network (AMMNet) - with a fresh perspective of
optimizing gradient updates. The proposed AMMNet introduces two core modules: a
cross-modal modulation enabling the interdependence of gradient flows between
modalities, and a customized adversarial training scheme leveraging dynamic
gradient competition. Specifically, the cross-modal modulation adaptively
re-calibrates the features to better excite representation potentials from each
single modality. The adversarial training employs a minimax game of evolving
gradients, with customized guidance to strengthen the generator's perception of
visual fidelity from both geometric completeness and semantic correctness.
Extensive experimental results demonstrate that AMMNet outperforms
state-of-the-art SSC methods by a large margin, providing a promising direction
for improving the effectiveness and generalization of SSC methods.

该研究提出了一种新的语义场景补全框架 - 对抗模态调制网络 (AMMNet)，通过优化梯度更新的方法，解决了单模态特征学习效果不佳和过拟合于有限数据集的两个问题，实验证明 AMMNet 在效果和泛化能力上明显优于最先进的语义场景补全方法。