Pioneering token-based works such as Chameleon and Emu3 have established a foundation for multimodal unification but face challenges of high training computational overhead and limited comprehension performance due to a lack of high-level semantics. In this paper, we introduce TokLIP, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens. TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics. Unlike previous approaches (e.g., VILA-U) that discretize high-level features, TokLIP disentangles training objectives for comprehension and generation, allowing the direct application of advanced VQ tokenizers without the need for tailored quantization operations. Our empirical results demonstrate that TokLIP achieves exceptional data efficiency, empowering visual tokens with high-level semantic understanding while enhancing low-level generative capacity, making it well-suited for autoregressive Transformers in both comprehension and generation tasks. The code and models are available at https://github.com/TencentARC/TokLIP.

本研究针对多模态统一中的训练计算开销高和理解性能受限的问题，提出了一种新颖的视觉令牌化方法TokLIP，通过语义化向量量化令牌并融合CLIP语义，进行端到端的多模态自回归训练。研究结果表明，TokLIP在数据效率方面表现出色，不仅赋予视觉令牌高层次的语义理解能力，还增强了低层次的生成能力，适用于自回归Transformer的理解和生成任务。

TokLIP：将视觉令牌与CLIP结合实现多模态理解与生成