Video Moment Retrieval (VMR) aims to retrieve temporal segments in untrimmed
videos corresponding to a given language query by constructing cross-modal
alignment strategies. However, these existing strategies are often sub-optimal
since they ignore the modality imbalance problem, \textit{i.e.}, the semantic
richness inherent in videos far exceeds that of a given limited-length
sentence. Therefore, in pursuit of better alignment, a natural idea is
enhancing the video modality to filter out query-irrelevant semantics, and
enhancing the text modality to capture more segment-relevant knowledge. In this
paper, we introduce Modal-Enhanced Semantic Modeling (MESM), a novel framework
for more balanced alignment through enhancing features at two levels. First, we
enhance the video modality at the frame-word level through word reconstruction.
This strategy emphasizes the portions associated with query words in
frame-level features while suppressing irrelevant parts. Therefore, the
enhanced video contains less redundant semantics and is more balanced with the
textual modality. Second, we enhance the textual modality at the
segment-sentence level by learning complementary knowledge from context
sentences and ground-truth segments. With the knowledge added to the query, the
textual modality thus maintains more meaningful semantics and is more balanced
with the video modality. By implementing two levels of MESM, the semantic
information from both modalities is more balanced to align, thereby bridging
the modality gap. Experiments on three widely used benchmarks, including the
out-of-distribution settings, show that the proposed framework achieves a new
start-of-the-art performance with notable generalization ability (e.g., 4.42%
and 7.69% average gains of R1@0.7 on Charades-STA and Charades-CG). The code
will be available at this https URL

通过提升视频模态和文本模态的特征，Modal-Enhanced Semantic Modeling（MESM）框架在视频短片检索中实现了更平衡的对齐，填补了形式上不平衡的模态差距。实验证明该框架在多个基准测试上取得了显著的泛化能力和最佳效果。

迈向平衡对齐：视频时刻检索的模态增强语义建模

Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video  Moment Retrieval

Existing multimodal conditional image synthesis (MCIS) methods generate
images conditioned on any combinations of various modalities that require all
of them must be exactly conformed, hindering the synthesis controllability and
leaving the potential of cross-modality under-exploited. To this end, we
propose to generate images conditioned on the compositions of multimodal
control signals, where modalities are imperfectly complementary, i.e., composed
multimodal conditional image synthesis (CMCIS). Specifically, we observe two
challenging issues of the proposed CMCIS task, i.e., the modality coordination
problem and the modality imbalance problem. To tackle these issues, we
introduce a Mixture-of-Modality-Tokens Transformer (MMoT) that adaptively fuses
fine-grained multimodal control signals, a multimodal balanced training loss to
stabilize the optimization of each modality, and a multimodal sampling guidance
to balance the strength of each modality control signal. Comprehensive
experimental results demonstrate that MMoT achieves superior performance on
both unimodal conditional image synthesis (UCIS) and MCIS tasks with
high-quality and faithful image synthesis on complex multimodal conditions. The
project website is available at this https URL

提出了一种基于混合多模态令牌变换器的组合多模态条件图像合成技术，解决了多模态控制信号协调问题和不平衡问题，实现了在复杂多模态条件下的高质量和忠实的图像合成。