PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address
the segmentation task challenges. To overcome the limitation of the LMM being
limited to textual output, PSALM incorporates a mask decoder and a
well-designed input schema to handle a variety of segmentation tasks. This
schema includes images, task instructions, conditional prompts, and mask
tokens, which enable the model to generate and classify segmentation masks
effectively. The flexible design of PSALM supports joint training across
multiple datasets and tasks, leading to improved performance and task
generalization. PSALM achieves superior results on several benchmarks, such as
RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive,
and further exhibits zero-shot capabilities on unseen tasks, such as
open-vocabulary segmentation, generalized referring expression segmentation and
video object segmentation, making a significant step towards a GPT moment in
computer vision. Through extensive experiments, PSALM demonstrates its
potential to transform the domain of image segmentation, leveraging the robust
visual understanding capabilities of LMMs as seen in natural language
processing. Code and models are available at this https URL

PSALM 是基于大型多模态模型（LMM）的强大扩展，用于解决分割任务的挑战，通过引入掩码解码器和精心设计的输入模式来处理各种分割任务，并且能够生成和分类分割掩码，支持多个数据集和任务的联合训练，具有卓越的性能和任务泛化能力，对图像分割领域具有潜力。

PSALM: 基于大型多模型的逐像素分割

PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model

Recently, prompt tuning \cite{lester2021power} has gradually become a new
paradigm for NLP, which only depends on the representation of the words by
freezing the parameters of pre-trained language models (PLMs) to obtain
remarkable performance on downstream tasks. It maintains the consistency of
Masked Language Model (MLM) \cite{devlin2018bert} task in the process of
pre-training, and avoids some issues that may happened during fine-tuning.
Naturally, we consider that the "[MASK]" tokens carry more useful information
than other tokens because the model combines with context to predict the masked
tokens. Among the current prompt tuning methods, there will be a serious
problem of random composition of the answer tokens in prediction when they
predict multiple words so that they have to map tokens to labels with the help
verbalizer. In response to the above issue, we propose a new \textbf{Pro}mpt
\textbf{Tu}ning based on "[\textbf{M}ASK]" (\textbf{Protum}) method in this
paper, which constructs a classification task through the information carried
by the hidden layer of "[MASK]" tokens and then predicts the labels directly
rather than the answer tokens. At the same time, we explore how different
hidden layers under "[MASK]" impact on our classification model on many
different data sets. Finally, we find that our \textbf{Protum} can achieve much
better performance than fine-tuning after continuous pre-training with less
time consumption. Our model facilitates the practical application of large
models in NLP.

本文提出了一种基于 'MASK' token 编码的 Prompt Tuning 方法 ——Protum，通过隐藏层信息构造分类任务并直接预测标签，相较于 fine-tuning，该方法在较短的时间内连续预训练后取得了更好的性能。

Protum: 一种基于 [MASK] 的新的提示调整方法

Protum: A New Method For Prompt Tuning Based on "[MASK]"

Recently very deep transformers have outperformed conventional bi-directional
long short-term memory networks by a large margin in speech recognition.
However, to put it into production usage, inference computation cost is still a
serious concern in real scenarios. In this paper, we study two different
non-autoregressive transformer structure for automatic speech recognition
(ASR): A-CMLM and A-FMLM. During training, for both frameworks, input tokens
fed to the decoder are randomly replaced by special mask tokens. The network is
required to predict the tokens corresponding to those mask tokens by taking
both unmasked context and input speech into consideration. During inference, we
start from all mask tokens and the network iteratively predicts missing tokens
based on partial results. We show that this framework can support different
decoding strategies, including traditional left-to-right. A new decoding
strategy is proposed as an example, which starts from the easiest predictions
to the most difficult ones. Results on Mandarin (Aishell) and Japanese (CSJ)
ASR benchmarks show the possibility to train such a non-autoregressive network
for ASR. Especially in Aishell, the proposed method outperformed the Kaldi ASR
system and it matches the performance of the state-of-the-art autoregressive
transformer with 7x speedup. Pretrained models and code will be made available
after publication.

本研究提出了两种不自回归变换器结构以解决自动语音识别中的推理计算成本问题，在训练期间，输入的标记被特殊的掩码标记随机替换，网络需要考虑未被掩盖的上下文和语音输入以预测与那些掩码标记相对应的标记。结果表明，该模型能够支持不同的解码策略，特别是在 Aishell 上，该方法的性能超过了 Kaldi ASR 系统，与最先进的自回归变换器的性能相当，而且速度提高了 7 倍。