We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels. To ground this prediction process in auto-regression, we customize a non-causal attention mask for the decoder, incorporating two key features: modeling tokens from different labels to be independent, and treating image tokens as a prefix. This masking mechanism inspires an efficient method - one-shot sampling - to simultaneously sample tokens of multiple labels in parallel and rank generated labels by their probabilities during inference. To further enhance the efficiency, we propose a simple strategy to construct a compact decoder by simply discarding the intermediate blocks of a pretrained language model. This approach yields a decoder that matches the full model's performance while being notably more efficient. The code is available at https://github.com/kaiyuyue/nxtp

通过将图像嵌入到文本令牌的自回归预测过程中，我们提出了一种将目标识别作为下一个令牌预测的方法。我们通过自定义非因果注意掩码来将预测过程与自回归相结合，其中包括将不同标签的令牌建模为独立，并将图像令牌视为前缀。我们提出了一种高效的一次性采样方法来同时并行采样多个标签的令牌，并在推理过程中通过它们的概率对生成的标签进行排名。为了进一步提高效率，我们提出了一个简单的策略，通过简单丢弃预训练语言模型的中间块来构建一个紧凑的解码器。这种方法在保持整体模型性能的同时具有显著的效率优势。