Simultaneous Machine Translation (SiMT) generates translation while reading
source tokens, essentially producing the target prefix based on the source
prefix. To achieve good performance, it leverages the relationship between
source and target prefixes to exact a policy to guide the generation of
translations. Although existing SiMT methods primarily focus on the
Encoder-Decoder architecture, we explore the potential of Decoder-only
architecture, owing to its superior performance in various tasks and its
inherent compatibility with SiMT. However, directly applying the Decoder-only
architecture to SiMT poses challenges in terms of training and inference. To
alleviate the above problems, we propose the first Decoder-only SiMT model,
named Decoder-only Streaming Transformer (DST). Specifically, DST separately
encodes the positions of the source and target prefixes, ensuring that the
position of the target prefix remains unaffected by the expansion of the source
prefix. Furthermore, we propose a Streaming Self-Attention (SSA) mechanism
tailored for the Decoder-only architecture. It is capable of obtaining
translation policy by assessing the sufficiency of input source information and
integrating with the soft-attention mechanism to generate translations.
Experiments demonstrate that our approach achieves state-of-the-art performance
on three translation tasks.

在这篇论文中，我们介绍了一种名为 Decoder-only Streaming Transformer (DST) 的 Decoder-only SiMT 模型，该模型通过编码源前缀和目标前缀的位置来确保目标前缀的位置不受源前缀扩展的影响，并通过 Streaming Self-Attention 机制获取翻译策略来生成翻译，实验证明我们的方法在三个翻译任务中实现了最先进的性能。

仅解码流式变换器用于同声传译

Decoder-only Streaming Transformer for Simultaneous Translation

The rapid progress of research aimed at interpreting the inner workings of
advanced language models has highlighted a need for contextualizing the
insights gained from years of work in this area. This primer provides a concise
technical introduction to the current techniques used to interpret the inner
workings of Transformer-based language models, focusing on the generative
decoder-only architecture. We conclude by presenting a comprehensive overview
of the known internal mechanisms implemented by these models, uncovering
connections across popular approaches and active research directions in this
area.

这篇论文提供了对 Transformer-based 语言模型内部工作进行解释的当前技术的简明技术介绍，重点讨论生成式只解码器架构。我们最后总结了这些模型实现的已知内部机制的综合概述，揭示了该领域中流行方法和活跃研究方向之间的联系。

基于 Transformer 的语言模型内部工作原理初探

A Primer on the Inner Workings of Transformer-based Language Models

Large language models (LLMs) have achieved remarkable success in the field of
natural language processing, enabling better human-computer interaction using
natural language. However, the seamless integration of speech signals into LLMs
has not been explored well. The "decoder-only" architecture has also not been
well studied for speech processing tasks. In this research, we introduce
Speech-LLaMA, a novel approach that effectively incorporates acoustic
information into text-based large language models. Our method leverages
Connectionist Temporal Classification and a simple audio encoder to map the
compressed acoustic features to the continuous semantic space of the LLM. In
addition, we further probe the decoder-only architecture for speech-to-text
tasks by training a smaller scale randomly initialized speech-LLaMA model from
speech-text paired data alone. We conduct experiments on multilingual
speech-to-text translation tasks and demonstrate a significant improvement over
strong baselines, highlighting the potential advantages of decoder-only models
for speech-to-text conversion.

该研究介绍了 Speech-LLaMA，一种将声学信息有效地整合到基于文本的大型语言模型中的新方法，并进一步探索了仅解码器架构在语音处理任务中的应用。

关于语音到文本和大型语言模型集成的仅解码器架构

On decoder-only architecture for speech-to-text and large language model  integration

We show that generating English Wikipedia articles can be approached as a
multi- document summarization of source documents. We use extractive
summarization to coarsely identify salient information and a neural abstractive
model to generate the article. For the abstractive model, we introduce a
decoder-only architecture that can scalably attend to very long sequences, much
longer than typical encoder- decoder architectures used in sequence
transduction. We show that this model can generate fluent, coherent
multi-sentence paragraphs and even whole Wikipedia articles. When given
reference documents, we show it can extract relevant factual information as
reflected in perplexity, ROUGE scores and human evaluations.

通过抽取式摘要识别重要信息并使用神经摘要模型生成文章及段落以及整个维基百科文章，特别是引入可扩展关注长序列的仅解码器结构，当给定参考文献时，该模型可以提取相关的事实信息。