Attention-based encoder-decoder (AED) models have shown impressive
performance in ASR. However, most existing AED methods neglect to
simultaneously leverage both acoustic and semantic features in decoder, which
is crucial for generating more accurate and informative semantic states. In
this paper, we propose an Acoustic and Semantic Cooperative Decoder (ASCD) for
ASR. In particular, unlike vanilla decoders that process acoustic and semantic
features in two separate stages, ASCD integrates them cooperatively. To prevent
information leakage during training, we design a Causal Multimodal Mask.
Moreover, a variant Semi-ASCD is proposed to balance accuracy and computational
cost. Our proposal is evaluated on the publicly available AISHELL-1 and
aidatatang_200zh datasets using Transformer, Conformer, and Branchformer as
encoders, respectively. The experimental results show that ASCD significantly
improves the performance by leveraging both the acoustic and semantic
information cooperatively.

该论文提出了一种声学和语义协同解码器 ASRD，可以同时利用声学和语义特征来提高自动语音识别的性能。通过引入因果多模态屏蔽，可以防止在训练期间的信息泄漏，并且还提出了一种改良版的半监督 ASCD 来平衡准确性和计算成本。实验结果表明，ASRD 可以显著提高 ASR 的准确性。

通过声学和语义合作解码重新思考多模态视角下的语音识别

Rethinking Speech Recognition with A Multimodal Perspective via Acoustic  and Semantic Cooperative Decoding

Recently, attention-based encoder-decoder models have been used extensively
in image captioning. Yet there is still great difficulty for the current
methods to achieve deep image understanding. In this work, we argue that such
understanding requires visual attention to correlated image regions and
semantic attention to coherent attributes of interest. Based on the
Transformer, to perform effective attention, we explore image captioning from a
cross-modal perspective and propose the Global-and-Local Information
Exploring-and-Distilling approach that explores and distills the source
information in vision and language. It globally provides the aspect vector, a
spatial and relational representation of images based on caption contexts,
through the extraction of salient region groupings and attribute collocations,
and locally extracts the fine-grained regions and attributes in reference to
the aspect vector for word selection. Our Transformer-based model achieves a
CIDEr score of 129.3 in offline COCO evaluation on the COCO testing set with
remarkable efficiency in terms of accuracy, speed, and parameter budget.

本研究提出基于 Transformer 的全局与局部信息探索与提炼方法，在跨模态视角下探究图像字幕生成，并在 COCO 测试集上取得了 129.3 的 CIDEr 分数。

探索和提取跨模态信息以用于图像字幕生成

Exploring and Distilling Cross-Modal Information for Image Captioning

We study the calibration of several state of the art neural machine
translation(NMT) systems built on attention-based encoder-decoder models. For
structured outputs like in NMT, calibration is important not just for reliable
confidence with predictions, but also for proper functioning of beam-search
inference. We show that most modern NMT models are surprisingly miscalibrated
even when conditioned on the true previous tokens. Our investigation leads to
two main reasons -- severe miscalibration of EOS (end of sequence marker) and
suppression of attention uncertainty. We design recalibration methods based on
these signals and demonstrate improved accuracy, better sequence-level
calibration, and more intuitive results from beam-search.

本文研究了几种基于注意力编码器 - 解码器模型的最新神经机器翻译 (NMT) 系统的校准。我们表明，大多数现代 NMT 模型即使在给定真正的先前标记的情况下，其校准水平也惊人地失准。 探讨了失误的两个原因：EOS 和注意力不确定性的严重失校。 基于这些信号，我们设计了重新校准方法，并展示了提高的准确性、更好的序列级校准和更直观的 beam-search 结果。