There is growing interest in searching for information from large video
corpora. Prior works have studied relevant tasks, such as text-based video
retrieval, moment retrieval, video summarization, and video captioning in
isolation, without an end-to-end setup that can jointly search from video
corpora and generate summaries. Such an end-to-end setup would allow for many
interesting applications, e.g., a text-based search that finds a relevant video
from a video corpus, extracts the most relevant moment from that video, and
segments the moment into important steps with captions. To address this, we
present the HiREST (HIerarchical REtrieval and STep-captioning) dataset and
propose a new benchmark that covers hierarchical information retrieval and
visual/textual stepwise summarization from an instructional video corpus.
HiREST consists of 3.4K text-video pairs from an instructional video dataset,
where 1.1K videos have annotations of moment spans relevant to text query and
breakdown of each moment into key instruction steps with caption and timestamps
(totaling 8.6K step captions). Our hierarchical benchmark consists of video
retrieval, moment retrieval, and two novel moment segmentation and step
captioning tasks. In moment segmentation, models break down a video moment into
instruction steps and identify start-end boundaries. In step captioning, models
generate a textual summary for each step. We also present starting point
task-specific and end-to-end joint baseline models for our new benchmark. While
the baseline models show some promising results, there still exists large room
for future improvement by the community. Project website:
this https URL

本研究介绍了 HiREST 数据集和基准测试，涵盖了从以下视频语料库中检索单元（瞬间）并分层检索高亮步骤的任务。HiREST 包括来自教学视频数据集的 3.4K 个文本视频对，其中 1.1K 个视频具有与文本查询相关的瞬间跨度注释，并将每个瞬间的重点说明步骤与时间戳（总计 8.6K 个步骤说明）的分解。本研究还为我们的新基准测试提供了一个起点任务特定的和端到端的联合基线模型。虽然基线模型显示出了一些有希望的结果，但仍然存在大量改进空间。

分层视频时刻检索和分步标题

Hierarchical Video-Moment Retrieval and Step-Captioning

Using class labels to represent class similarity is a typical approach to
training deep hashing systems for retrieval; samples from the same or different
classes take binary 1 or 0 similarity values. This similarity does not model
the full rich knowledge of semantic relations that may be present between data
points. In this work we build upon the idea of using semantic hierarchies to
form distance metrics between all available sample labels; for example cat to
dog has a smaller distance than cat to guitar. We combine this type of semantic
distance into a loss function to promote similar distances between the deep
neural network embeddings. We also introduce an empirical Kullback-Leibler
divergence loss term to promote binarization and uniformity of the embeddings.
We test the resulting SHREWD method and demonstrate improvements in
hierarchical retrieval scores using compact, binary hash codes instead of real
valued ones, and show that in a weakly supervised hashing setting we are able
to learn competitively without explicitly relying on class labels, but instead
on similarities between labels.

本研究基于语义层次结构，对样本标签间的距离进行度量，并结合该距离构建损失函数以提高深度神经网络嵌入的相似性，同时引入经验 Kullback-Leibler 散度损失项，以提高嵌入的二值化和均匀性，从而实现利用紧凑二进制哈希码而非实值哈希码进行分层检索，且在弱监督哈希设置下，能够在不显式依赖类标签的情况下学习相似度。

基于语义层级的关系嵌入在弱监督深度哈希中的应用

SHREWD: Semantic Hierarchy-based Relational Embeddings for  Weakly-supervised Deep Hashing

Deep hashing models have been proposed as an efficient method for large-scale
similarity search. However, most existing deep hashing methods only utilize
fine-level labels for training while ignoring the natural semantic hierarchy
structure. This paper presents an effective method that preserves the classwise
similarity of full-level semantic hierarchy for large-scale image retrieval.
Experiments on two benchmark datasets show that our method helps improve the
fine-level retrieval performance. Moreover, with the help of the semantic
hierarchy, it can produce significantly better binary codes for hierarchical
retrieval, which indicates its potential of providing more user-desired
retrieval results.

本研究提出了一种有效的方法，以保留完整语义层次结构的类别相似性，以提高大规模图像检索的精细级别性能并生成更好的二进制代码。