3D visual grounding is the ability to localize objects in 3D scenes
conditioned by utterances. Most existing methods devote the referring head to
localize the referred object directly, causing failure in complex scenarios. In
addition, it does not illustrate how and why the network reaches the final
decision. In this paper, we address this question Can we design an
interpretable 3D visual grounding framework that has the potential to mimic the
human perception system?. To this end, we formulate the 3D visual grounding
problem as a sequence-to-sequence task by first predicting a chain of anchors
and then the final target. Interpretability not only improves the overall
performance but also helps us identify failure cases. Following the chain of
thoughts approach enables us to decompose the referring task into interpretable
intermediate steps, boosting the performance and making our framework extremely
data-efficient. Moreover, our proposed framework can be easily integrated into
any existing architecture. We validate our approach through comprehensive
experiments on the Nr3D, Sr3D, and Scanrefer benchmarks and show consistent
performance gains compared to existing methods without requiring manually
annotated data. Furthermore, our proposed framework, dubbed CoT3DRef, is
significantly data-efficient, whereas on the Sr3D dataset, when trained only on
10% of the data, we match the SOTA performance that trained on the entire data.

设计一种可解释的 3D 视觉定位框架，通过预测一系列锚点和最终目标，将 3D 视觉定位问题形式化为序列到序列任务，将指称任务分解为可解释的中间步骤，从而提高性能并极大地提高数据效率。

CoT3DRef: 链状思考数据高效 3D 视觉定位

CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding

Audio-visual speech recognition (AVSR) attracts a surge of research interest
recently by leveraging multimodal signals to understand human speech.
Mainstream approaches addressing this task have developed sophisticated
architectures and techniques for multi-modality fusion and representation
learning. However, the natural heterogeneity of different modalities causes
distribution gap between their representations, making it challenging to fuse
them. In this paper, we aim to learn the shared representations across
modalities to bridge their gap. Different from existing similar methods on
other multimodal tasks like sentiment analysis, we focus on the temporal
contextual dependencies considering the sequence-to-sequence task setting of
AVSR. In particular, we propose an adversarial network to refine frame-level
modality-invariant representations (MIR-GAN), which captures the commonality
across modalities to ease the subsequent multimodal fusion process. Extensive
experiments on public benchmarks LRS3 and LRS2 show that our approach
outperforms the state-of-the-arts.

本文提出了一种基于对抗网络的方法（MIR-GAN），旨在跨模态学习共享的表征，以减轻后续的多模态融合过程，并在公共基准 LRS3 和 LRS2 上得到了比现有技术更好的表现。

MIR-GAN：用对抗网络提炼用于音视频语音识别的帧层次模态不变表示

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with  Adversarial Network for Audio-Visual Speech Recognition

The sliding window approach provides an elegant way to handle contexts of
sizes larger than the Transformer's input window, for tasks like language
modeling. Here we extend this approach to the sequence-to-sequence task of
document parsing. For this, we exploit recent progress in transition-based
parsing to implement a parser with synchronous sliding windows over source and
target. We develop an oracle and a parser for document-level AMR by expanding
on Structured-BART such that it leverages source-target alignments and
constrains decoding to guarantee synchronicity and consistency across
overlapping windows. We evaluate our oracle and parser using the Abstract
Meaning Representation (AMR) parsing 3.0 corpus. On the Multi-Sentence
development set of AMR 3.0, we show that our transition oracle loses only 8\%
of the gold cross-sentential links despite using a sliding window. In practice,
this approach also results in a high-quality document-level parser with
manageable memory requirements. Our proposed system performs on par with the
state-of-the-art pipeline approach for document-level AMR parsing task on
Multi-Sentence AMR 3.0 corpus while maintaining sentence-level parsing
performance.

利用滑动窗口和转移依存句法分析技术提出了一种同步滑动窗口方法，用于解决多句文档的序列到序列任务，其具有很高的性能表现，并不损失黄金跨句链接。

幻灯片、约束、解析、重复：用于文档 AMR 解析的同步滑动窗口

Slide, Constrain, Parse, Repeat: Synchronous SlidingWindows for Document  AMR Parsing

Pre-trained language models have shown remarkable success in improving
various downstream NLP tasks due to their ability to capture dependencies in
textual data and generate natural responses. In this paper, we leverage the
power of pre-trained language models for improving video-grounded dialogue,
which is very challenging and involves complex features of different dynamics:
(1) Video features which can extend across both spatial and temporal
dimensions; and (2) Dialogue features which involve semantic dependencies over
multiple dialogue turns. We propose a framework by extending GPT-2 models to
tackle these challenges by formulating video-grounded dialogue tasks as a
sequence-to-sequence task, combining both visual and textual representation
into a structured sequence, and fine-tuning a large pre-trained GPT-2 network.
Our framework allows fine-tuning language models to capture dependencies across
multiple modalities over different levels of information: spatio-temporal level
in video and token-sentence level in dialogue context. We achieve promising
improvement on the Audio-Visual Scene-Aware Dialogues (AVSD) benchmark from
DSTC7, which supports a potential direction in this line of research.

本文提出了一种基于 GPT-2 模型的框架，将视频与文本表示结合成连续、有结构的序列，并利用其 fine-tuning 能力来解决视频对话中的挑战，从而在 Audio-Visual Scene-Aware Dialogues 基准测试中取得了显著的改进。