Speech Recognition builds a bridge between the multimedia streaming
(audio-only, visual-only or audio-visual) and the corresponding text
transcription. However, when training the specific model of new domain, it
often gets stuck in the lack of new-domain utterances, especially the labeled
visual utterances. To break through this restriction, we attempt to achieve
zero-shot modality transfer by maintaining the multi-modality alignment in
phoneme space learned with unlabeled multimedia utterances in the high resource
domain during the pre-training \cite{shi2022learning}, and propose a training
system Open-modality Speech Recognition (\textbf{OpenSR}) that enables the
models trained on a single modality (e.g., audio-only) applicable to more
modalities (e.g., visual-only and audio-visual). Furthermore, we employ a
cluster-based prompt tuning strategy to handle the domain shift for the
scenarios with only common words in the new domain utterances. We demonstrate
that OpenSR enables modality transfer from one to any in three different
settings (zero-, few- and full-shot), and achieves highly competitive zero-shot
performance compared to the existing few-shot and full-shot lip-reading
methods. To the best of our knowledge, OpenSR achieves the state-of-the-art
performance of word error rate in LRS2 on audio-visual speech recognition and
lip-reading with 2.7\% and 25.0\%, respectively. The code and demo are
available at this https URL

本研究提出 OpenSR，通过维护来自高资源领域的未标记多媒体话语中音位空间中的多模态对齐，实现零样例模态转换，进而实现应用于不同模态的模型训练并采用基于聚类的提示调整策略来处理新域话语中的领域移位，结果表明 OpenSR 在三种不同设置下实现了模态转换，并在音视频语音识别和唇语识别方面取得了最先进的性能。

OpenSR: 通过维护多模态对齐实现开放式语音识别

OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality  Alignment

In this paper we present an approach for localizing steps of procedural
activities in narrated how-to videos. To deal with the scarcity of labeled data
at scale, we source the step descriptions from a language knowledge base
(wikiHow) containing instructional articles for a large variety of procedural
tasks. Without any form of manual supervision, our model learns to temporally
ground the steps of procedural articles in how-to videos by matching three
modalities: frames, narrations, and step descriptions. Specifically, our method
aligns steps to video by fusing information from two distinct pathways: i) {\em
direct} alignment of step descriptions to frames, ii) {\em indirect} alignment
obtained by composing steps-to-narrations with narrations-to-video
correspondences. Notably, our approach performs global temporal grounding of
all steps in an article at once by exploiting order information, and is trained
with step pseudo-labels which are iteratively refined and aggressively
filtered. In order to validate our model we introduce a new evaluation
benchmark -- HT-Step -- obtained by manually annotating a 124-hour subset of
HowTo100M\footnote{A test server is accessible at
https://eval.ai/web/challenges/challenge-page/2082.} with steps sourced
from wikiHow articles. Experiments on this benchmark as well as zero-shot
evaluations on CrossTask demonstrate that our multi-modality alignment yields
dramatic gains over several baselines and prior works. Finally, we show that
our inner module for matching narration-to-video outperforms by a large margin
the state of the art on the HTM-Align narration-video alignment benchmark.

本文提出了一种无需手动监督的方法来将『如何做』视频中的步骤与 wikiHow 等语言知识库中的指导文章进行对齐，并采用多模态对齐进行全局时间定位，验证结果表明与现有方法相比具有明显优势。

通过叙述学习将说明文章与视频联系起来

Learning to Ground Instructional Articles in Videos through Narrations

Most existing RGB-based trackers target low frame rate benchmarks of around
30 frames per second. This setting restricts the tracker's functionality in the
real world, especially for fast motion. Event-based cameras as bioinspired
sensors provide considerable potential for high frame rate tracking due to
their high temporal resolution. However, event-based cameras cannot offer
fine-grained texture information like conventional cameras. This unique
complementarity motivates us to combine conventional frames and events for high
frame rate object tracking under various challenging conditions. Inthispaper,
we propose an end-to-end network consisting of multi-modality alignment and
fusion modules to effectively combine meaningful information from both
modalities at different measurement rates. The alignment module is responsible
for cross-style and cross-frame-rate alignment between frame and event
modalities under the guidance of the moving cues furnished by events. While the
fusion module is accountable for emphasizing valuable features and suppressing
noise information by the mutual complement between the two modalities.
Extensive experiments show that the proposed approach outperforms
state-of-the-art trackers by a significant margin in high frame rate tracking.
With the FE240hz dataset, our approach achieves high frame rate tracking up to
240Hz.

该论文提出了使用多模方式结合 RGB-based trackers 和 event-based cameras 进行 high frame rate tracking 的方法，并应用 multi-modality alignment 和 fusion modules 对两种模式的信息进行融合。在 FE240hz 数据集上，该方法表现优异，实现了高达 240Hz 的高帧率追踪。