Recent research shows a big convergence in model architecture, training
objectives, and inference methods across various tasks for different
modalities. In this paper, we propose VioLA, a single auto-regressive
Transformer decoder-only network that unifies various cross-modal tasks
involving speech and text, such as speech-to-text, text-to-text,
text-to-speech, and speech-to-speech tasks, as a conditional codec language
model task via multi-task learning framework. To accomplish this, we first
convert all the speech utterances to discrete tokens (similar to the textual
data) using an offline neural codec encoder. In such a way, all these tasks are
converted to token-based sequence conversion problems, which can be naturally
handled with one conditional language model. We further integrate task IDs
(TID) and language IDs (LID) into the proposed model to enhance the modeling
capability of handling different languages and tasks. Experimental results
demonstrate that the proposed VioLA model can support both single-modal and
cross-modal tasks well, and the decoder-only model achieves a comparable and
even better performance than the strong baselines.

本研究提出了 VioLA 模型，它是一个单自回归 Transformer 解码器网络，通过多任务学习框架，将涉及语音和文本的各种跨模态任务统一为条件编解码器语言模型任务。该模型通过离线神经编解码器将所有语音话语转换为离散标记，进而将所有任务转换为基于标记的序列转换问题，进而自然地使用一个条件语言模型处理。研究表明，VioLA 模型可良好地支持单模和跨模任务，并且解码器模型优于强基线性能。

VioLA: 面向语音识别、合成和翻译的统一编解码器语言模型

VioLA: Unified Codec Language Models for Speech Recognition, Synthesis,  and Translation

End-to-end scene text spotting has made significant progress due to its
intrinsic synergy between text detection and recognition. Previous methods
commonly regard manual annotations such as horizontal rectangles, rotated
rectangles, quadrangles,and polygons as a prerequisite, which are much more
expensive than using single-point. For the first time, we demonstrate that
training scene text spotting models can be achieved with an extremely low-cost
single-point annotation by the proposed framework, termed SPTS v2. SPTS v2
reserves the advantage of the auto-regressive Transformer with an Instance
Assignment Decoder (IAD) through sequentially predicting the center points of
all text instances inside the same predicting sequence, while with a Parallel
Recognition Decoder (PRD) for text recognition in parallel. These two decoders
share the same parameters and are interactively connected with a simple but
effective information transmission process to pass the gradient and
information. Comprehensive experiments on various existing benchmark datasets
demonstrate the SPTS v2 can outperform previous state-of-the-art single-point
text spotters with fewer parameters while achieving 14x faster inference speed.
Most importantly, within the scope of our SPTS v2, extensive experiments
further reveal an important phenomenon that single-point serves as the optimal
setting for the scene text spotting compared to non-point, rectangular bounding
box, and polygonal bounding box. Such an attempt provides a significant
opportunity for scene text spotting applications beyond the realms of existing
paradigms. Code is available at this https URL.

本文提出了 SPTS v2 框架，采用单点注释作为代替传统文本检测与识别的昂贵手工注释方法。采用基于自回归 Transformer 的 IAD 对同一预测序列内所有文本实例的中心点进行预测，采用 PRD 进行文本识别。详细实验结果证明该方法可超越现有单点文本定位器的性能而实现 14 倍更快的推断速度。同时，实验还进一步展示了单点注释比非点注释和多边形边界框更为适合场景文本识别。

SPTS v2：单点场景文本检测

SPTS v2: Single-Point Scene Text Spotting

We address the problem of action-conditioned generation of human motion
sequences. Existing work falls into two categories: forecast models conditioned
on observed past motions, or generative models conditioned on action labels and
duration only. In contrast, we generate motion conditioned on observations of
arbitrary length, including none. To solve this generalized problem, we propose
PoseGPT, an auto-regressive transformer-based approach which internally
compresses human motion into quantized latent sequences. An auto-encoder first
maps human motion to latent index sequences in a discrete space, and
vice-versa. Inspired by the Generative Pretrained Transformer (GPT), we propose
to train a GPT-like model for next-index prediction in that space; this allows
PoseGPT to output distributions on possible futures, with or without
conditioning on past motion. The discrete and compressed nature of the latent
space allows the GPT-like model to focus on long-range signal, as it removes
low-level redundancy in the input signal. Predicting discrete indices also
alleviates the common pitfall of predicting averaged poses, a typical failure
case when regressing continuous values, as the average of discrete targets is
not a target itself. Our experimental results show that our proposed approach
achieves state-of-the-art results on HumanAct12, a standard but small scale
dataset, as well as on BABEL, a recent large scale MoCap dataset, and on GRAB,
a human-object interactions dataset.

提出了一种基于 PoseGPT 的自回归变换器的方法，该方法可以生成人体运动序列，利用离散的潜在空间使 GPT-like 模型集中于长程信号预测，获得了在几个数据集上实现最先进的结果。