The rise in loosely-structured data available through text, images, and other
modalities has called for new ways of querying them. Multimedia Information
Retrieval has filled this gap and has witnessed exciting progress in recent
years. Tasks such as search and retrieval of extensive multimedia archives have
undergone massive performance improvements, driven to a large extent by recent
developments in multimodal deep learning. However, methods in this field remain
limited in the kinds of queries they support and, in particular, their
inability to answer database-like queries. For this reason, inspired by recent
work on neural databases, we propose a new framework, which we name Multimodal
Neural Databases (MMNDBs). MMNDBs can answer complex database-like queries that
involve reasoning over different input modalities, such as text and images, at
scale. In this paper, we present the first architecture able to fulfill this
set of requirements and test it with several baselines, showing the limitations
of currently available models. The results show the potential of these new
techniques to process unstructured data coming from different modalities,
paving the way for future research in the area. Code to replicate the
experiments will be released at
this https URL

本文提出了一种新的框架，即多模态神经数据库（MMNDBs），用于回答涉及不同输入模式（如文本和图像）的复杂类似数据库的查询，通过多媒体信息检索和多模态深度学习技术的结合，MMNDBs 能够处理来自不同模式的非结构化数据，为该领域的未来研究铺平了道路。

多模态神经数据库

Multimodal Neural Databases

Temporal sentence grounding (TSG) is an important yet challenging task in
multimedia information retrieval. Although previous TSG methods have achieved
decent performance, they tend to capture the selection biases of frequently
appeared video-query pairs in the dataset rather than present robust multimodal
reasoning abilities, especially for the rarely appeared pairs. In this paper,
we study the above issue of selection biases and accordingly propose a
Debiasing-TSG (D-TSG) model to filter and remove the negative biases in both
vision and language modalities for enhancing the model generalization ability.
Specifically, we propose to alleviate the issue from two perspectives: 1)
Feature distillation. We built a multi-modal debiasing branch to firstly
capture the vision and language biases, and then apply a bias identification
module to explicitly recognize the true negative biases and remove them from
the benign multi-modal representations. 2) Contrastive sample generation. We
construct two types of negative samples to enforce the model to accurately
learn the aligned multi-modal semantics and make complete semantic reasoning.
We apply the proposed model to both commonly and rarely appeared TSG cases, and
demonstrate its effectiveness by achieving the state-of-the-art performance on
three benchmark datasets (ActivityNet Caption, TACoS, and Charades-STA).

本文提出了一种消除多媒体信息检索中选择偏差的 Debiasing-TSG（D-TSG）模型，实现在常见和罕见情况下的 Temporal sentence grounding，在三个基准数据集上取得了最先进的表现。