Visually grounded speech models link speech to images. We extend this
connection by linking images to text via an existing image captioning system,
and as a result gain the ability to map speech audio directly to text. This
approach can be used for speech translation with just images by having the
audio in a different language from the generated captions. We investigate such
a system on a real low-resource language, Yor\`ub\'a, and propose a
Yor\`ub\'a-to-English speech translation model that leverages pretrained
components in order to be able to learn in the low-resource regime. To limit
overfitting, we find that it is essential to use a decoding scheme that
produces diverse image captions for training. Results show that the predicted
translations capture the main semantics of the spoken audio, albeit in a
simpler and shorter form.

通过现有的图像字幕系统，将图像与文本相连接，实现语音音频与文本之间的直接映射，从而扩展了视觉语音模型的连接方式。本研究在一个低资源语言 Yorùbá 上，提出了一种 Yorùbá- 英语语音翻译模型，利用预训练组件以实现在低资源环境中的学习，并通过使用能产生多样的图像字幕的解码方案来限制过拟合现象。结果显示预测的翻译捕捉到了口头音频的主要语义，尽管形式上更简单且更简短。

仅使用图像进行语音翻译

Translating speech with just images

Multi-scale architecture, including hierarchical vision transformer, has been
commonly applied to high-resolution semantic segmentation to deal with
computational complexity with minimum performance loss. In this paper, we
propose a novel decoding scheme for semantic segmentation in this regard, which
takes multi-level features from the encoder with multi-scale architecture. The
decoding scheme based on a multi-level vision transformer aims to achieve not
only reduced computational expense but also higher segmentation accuracy, by
introducing successive cross-attention in aggregation of the multi-level
features. Furthermore, a way to enhance the multi-level features by the
aggregated semantics is proposed. The effort is focused on maintaining the
contextual consistency from the perspective of attention allocation and brings
improved performance with significantly lower computational cost. Set of
experiments on popular datasets demonstrates superiority of the proposed scheme
to the state-of-the-art semantic segmentation models in terms of computational
cost without loss of accuracy, and extensive ablation studies prove the
effectiveness of ideas proposed.

通过多级视觉变换编码器中的多级特征，本文提出了一种新颖的语义分割解码方案，旨在降低计算开销、提高分割准确性，并通过聚合后的语义增强多级特征，从注意力分配的角度保持上下文一致性，大幅降低计算成本。实验结果表明该方案在计算成本上优于现有语义分割模型，且没有准确性损失；深入的消融研究证明了所提出思路的有效性。