Recent methods for visual question answering rely on large-scale annotated
datasets. Manual annotation of questions and answers for videos, however, is
tedious, expensive and prevents scalability. In this work, we propose to avoid
manual annotation and generate a large-scale training dataset for video
question answering making use of automatic cross-modal supervision. We leverage
a question generation transformer trained on text data and use it to generate
question-answer pairs from transcribed video narrations. Given narrated videos,
we then automatically generate the HowToVQA69M dataset with 69M
video-question-answer triplets. To handle the open vocabulary of diverse
answers in this dataset, we propose a training procedure based on a contrastive
loss between a video-question multi-modal transformer and an answer
transformer. We introduce the zero-shot VideoQA task and show excellent
results, in particular for rare answers. Furthermore, we demonstrate our method
to significantly outperform the state of the art on MSRVTT-QA, MSVD-QA,
ActivityNet-QA and How2QA. Finally, for a detailed evaluation we introduce
iVQA, a new VideoQA dataset with reduced language biases and high-quality
redundant manual annotations. Our code, datasets and trained models are
available at this https URL

本研究提出了一种使用自动跨模态监督和问题生成转换器生成问题和答案对的方法来生成大规模视频问答训练数据集，并提出了一种基于对比损失的训练程序来处理其多样的答案类型。通过实验证明，在 MSRVTT-QA、MSVD-QA、ActivityNet-QA 和 How2QA 等多个任务上，该方法明显优于现有技术。

从数百万个叙述视频中学习回答问题

Just Ask: Learning to Answer Questions from Millions of Narrated Videos

Many people search for foreground objects to use when editing images. While
existing methods can retrieve candidates to aid in this, they are constrained
to returning objects that belong to a pre-specified semantic class. We instead
propose a novel problem of unconstrained foreground object (UFO) search and
introduce a solution that supports efficient search by encoding the background
image in the same latent space as the candidate foreground objects. A key
contribution of our work is a cost-free, scalable approach for creating a
large-scale training dataset with a variety of foreground objects of differing
semantic categories per image location. Quantitative and human-perception
experiments with two diverse datasets demonstrate the advantage of our UFO
search solution over related baselines.

本文提出了一种解决图像编辑中寻找前景对象的技术 —— 无约束前景物体搜索，并介绍了一种支持通过将背景图像与前景对象进行编码来进行高效搜索的解决方案。通过创建具有不同语义类别的多个前景对象的大型训练数据集，我们的无约束前景物体搜索解决方案优于相关基线，这在两个不同的数据集上进行了定量和人类感知实验的证实。