Seeing clearly with high resolution is a foundation of Large Multimodal
Models (LMMs), which has been proven to be vital for visual perception and
reasoning. Existing works usually employ a straightforward resolution upscaling
method, where the image consists of global and local branches, with the latter
being the sliced image patches but resized to the same resolution as the
former. This means that higher resolution requires more local patches,
resulting in exorbitant computational expenses, and meanwhile, the dominance of
local image tokens may diminish the global context. In this paper, we dive into
the problems and propose a new framework as well as an elaborate optimization
strategy. Specifically, we extract contextual information from the global view
using a mixture of adapters, based on the observation that different adapters
excel at different tasks. With regard to local patches, learnable query
embeddings are introduced to reduce image tokens, the most important tokens
accounting for the user question will be further selected by a similarity-based
selector. Our empirical results demonstrate a `less is more' pattern, where
\textit{utilizing fewer but more informative local image tokens leads to
improved performance}. Besides, a significant challenge lies in the training
strategy, as simultaneous end-to-end training of the global mining block and
local compression block does not yield optimal results. We thus advocate for an
alternating training way, ensuring balanced learning between global and local
aspects. Finally, we also introduce a challenging dataset with high
requirements for image detail, enhancing the training of the local compression
layer. The proposed method, termed LMM with Sophisticated Tasks, Local image
compression, and Mixture of global Experts (SliME), achieves leading
performance across various benchmarks with only 2 million training data.

高分辨率是大型多模式模型（LMMs）的基础，本论文提出了一种新的框架和优化策略，通过混合适配器从全局视图中提取上下文信息，并引入可学习的查询嵌入来减少图像标记，同时通过相似性选择器选择用户问题的关键标记，实现更好的性能表现。此外，通过交替训练的方式平衡学习全局和局部方面，并引入高要求图像细节的数据集来增强局部压缩层的训练，提出的方法在各项基准测试中表现出优异性能。