The tokenizer, as one of the fundamental components of large models, has long
been overlooked or even misunderstood in visual tasks. One key factor of the
great comprehension power of the large language model is that natural language
tokenizers utilize meaningful words or subwords as the basic elements of
language. In contrast, mainstream visual tokenizers, represented by patch-based
methods such as Patch Embed, rely on meaningless rectangular patches as basic
elements of vision, which cannot serve as effectively as words or subwords in
language. Starting from the essence of the tokenizer, we defined semantically
independent regions (SIRs) for vision. We designed a simple HOmogeneous visual
tOKenizer: HOOK. HOOK mainly consists of two modules: the Object Perception
Module (OPM) and the Object Vectorization Module (OVM). To achieve homogeneity,
the OPM splits the image into 4*4 pixel seeds and then utilizes the attention
mechanism to perceive SIRs. The OVM employs cross-attention to merge seeds
within the same SIR. To achieve adaptability, the OVM defines a variable number
of learnable vectors as cross-attention queries, allowing for the adjustment of
token quantity. We conducted experiments on the NWPU-RESISC45, WHU-RS19
classification dataset, and GID5 segmentation dataset for sparse and dense
tasks. The results demonstrate that the visual tokens obtained by HOOK
correspond to individual objects, which demonstrates homogeneity. HOOK
outperformed Patch Embed by 6\% and 10\% in the two tasks and achieved
state-of-the-art performance compared to the baselines used for comparison.
Compared to Patch Embed, which requires more than one hundred tokens for one
image, HOOK requires only 6 and 8 tokens for sparse and dense tasks,
respectively, resulting in efficiency improvements of 1.5 to 2.8 times. The
code is available at this https URL

基于视觉感知的齐质化可视化分词器 HOOK 在稀疏和密集任务中使用较少的令牌，以实现与 Patch Embed 的性能改进和效率提高。