Information Retriever (IR) aims to find the relevant documents (e.g.
snippets, passages, and articles) to a given query at large scale. IR plays an
important role in many tasks such as open domain question answering and
dialogue systems, where external knowledge is needed. In the past, searching
algorithms based on term matching have been widely used. Recently, neural-based
algorithms (termed as neural retrievers) have gained more attention which can
mitigate the limitations of traditional methods. Regardless of the success
achieved by neural retrievers, they still face many challenges, e.g. suffering
from a small amount of training data and failing to answer simple
entity-centric questions. Furthermore, most of the existing neural retrievers
are developed for pure-text query. This prevents them from handling
multi-modality queries (i.e. the query is composed of textual description and
images). This proposal has two goals. First, we introduce methods to address
the abovementioned issues of neural retrievers from three angles, new model
architectures, IR-oriented pretraining tasks, and generating large scale
training data. Second, we identify the future research direction and propose
potential corresponding solution.

本篇论文提出三个方面的方法，包括新的模型架构、IR 导向的预训练任务、生成大规模训练数据，以解决神经检索器面临的问题，并探讨了多模式查询的未来研究方向和相应的解决方法。

神经检索器与超越：论文提案

Neural Retriever and Go Beyond: A Thesis Proposal

The target of automatic video summarization is to create a short skim of the
original long video while preserving the major content/events. There is a
growing interest in the integration of user queries into video summarization or
query-driven video summarization. This video summarization method predicts a
concise synopsis of the original video based on the user query, which is
commonly represented by the input text. However, two inherent problems exist in
this query-driven way. First, the text query might not be enough to describe
the exact and diverse needs of the user. Second, the user cannot edit once the
summaries are produced, while we assume the needs of the user should be subtle
and need to be adjusted interactively. To solve these two problems, we propose
IntentVizor, an interactive video summarization framework guided by generic
multi-modality queries. The input query that describes the user's needs are not
limited to text but also the video snippets. We further represent these
multi-modality finer-grained queries as user `intent', which is interpretable,
interactable, editable, and can better quantify the user's needs. In this
paper, we use a set of the proposed intents to represent the user query and
design a new interactive visual analytic interface. Users can interactively
control and adjust these mixed-initiative intents to obtain a more satisfying
summary through the interface. Also, to improve the summarization quality via
video understanding, a novel Granularity-Scalable Ego-Graph Convolutional
Networks (GSE-GCN) is proposed. We conduct our experiments on two benchmark
datasets. Comparisons with the state-of-the-art methods verify the
effectiveness of the proposed framework. Code and dataset are available at
this https URL

本文介绍 IntentVizor，一种基于多模态查询的交互式视频摘要框架。通过使用我们所提出的一系列意图，我们设计了一个新型交互式视觉分析界面，并使用 GSE-GCN 来提高视频理解的摘要质量。我们针对两个基准数据集进行了实验，并与现有方法进行了比较，验证了该框架的有效性。