Seeing clearly with high resolution is a foundation of Large Multimodal
Models (LMMs), which has been proven to be vital for visual perception and
reasoning. Existing works usually employ a straightforward resolution upscaling
method, where the image consists of global and local branches, with the latter
being the sliced image patches but resized to the same resolution as the
former. This means that higher resolution requires more local patches,
resulting in exorbitant computational expenses, and meanwhile, the dominance of
local image tokens may diminish the global context. In this paper, we dive into
the problems and propose a new framework as well as an elaborate optimization
strategy. Specifically, we extract contextual information from the global view
using a mixture of adapters, based on the observation that different adapters
excel at different tasks. With regard to local patches, learnable query
embeddings are introduced to reduce image tokens, the most important tokens
accounting for the user question will be further selected by a similarity-based
selector. Our empirical results demonstrate a `less is more' pattern, where
\textit{utilizing fewer but more informative local image tokens leads to
improved performance}. Besides, a significant challenge lies in the training
strategy, as simultaneous end-to-end training of the global mining block and
local compression block does not yield optimal results. We thus advocate for an
alternating training way, ensuring balanced learning between global and local
aspects. Finally, we also introduce a challenging dataset with high
requirements for image detail, enhancing the training of the local compression
layer. The proposed method, termed LMM with Sophisticated Tasks, Local image
compression, and Mixture of global Experts (SliME), achieves leading
performance across various benchmarks with only 2 million training data.

高分辨率是大型多模式模型（LMMs）的基础，本论文提出了一种新的框架和优化策略，通过混合适配器从全局视图中提取上下文信息，并引入可学习的查询嵌入来减少图像标记，同时通过相似性选择器选择用户问题的关键标记，实现更好的性能表现。此外，通过交替训练的方式平衡学习全局和局部方面，并引入高要求图像细节的数据集来增强局部压缩层的训练，提出的方法在各项基准测试中表现出优异性能。

超高像素大型多模态模型

Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models

Temporal Knowledge Graph (TKG) reasoning is based on historical information
to predict the future. Therefore, parsing and mining historical information is
key to predicting the future. Most existing methods fail to concurrently
address and comprehend historical information from both global and local
perspectives. Neglecting the global view might result in overlooking
macroscopic trends and patterns, while ignoring the local view can lead to
missing critical detailed information. Additionally, some methods do not focus
on learning from high-frequency repeating events, which means they may not
fully grasp frequently occurring historical events. To this end, we propose the
\textbf{R}epetitive-\textbf{L}ocal-\textbf{G}lobal History
\textbf{Net}work(RLGNet). We utilize a global history encoder to capture the
overarching nature of historical information. Subsequently, the local history
encoder provides information related to the query timestamp. Finally, we employ
the repeating history encoder to identify and learn from frequently occurring
historical events. In the evaluation on six benchmark datasets, our approach
generally outperforms existing TKG reasoning models in multi-step and
single-step reasoning tasks.

通过使用	extbf {R} epetitive-	extbf {L} ocal-	extbf {G} lobal History 	extbf {Net} work (RLGNet)，我们提出了一个基于历史信息的方法，既考虑了全局视角又关注了局部视角，同时能够学习频繁发生的历史事件，并在多步和单步推理任务中显著优于现有的时间知识图推理模型。

RLGNet: 重复 - 本地 - 全局历史网络用于时态知识图推理

RLGNet: Repeating-Local-Global History Network for Temporal Knowledge  Graph Reasoning

Most of the few-shot learning methods learn to transfer knowledge from
datasets with abundant labeled data (i.e., the base set). From the perspective
of class space on base set, existing methods either focus on utilizing all
classes under a global view by normal pretraining, or pay more attention to
adopt an episodic manner to train meta-tasks within few classes in a local
view. However, the interaction of the two views is rarely explored. As the two
views capture complementary information, we naturally think of the
compatibility of them for achieving further performance gains. Inspired by the
mutual learning paradigm and binocular parallax, we propose a unified
framework, namely Binocular Mutual Learning (BML), which achieves the
compatibility of the global view and the local view through both intra-view and
cross-view modeling. Concretely, the global view learns in the whole class
space to capture rich inter-class relationships. Meanwhile, the local view
learns in the local class space within each episode, focusing on matching
positive pairs correctly. In addition, cross-view mutual interaction further
promotes the collaborative learning and the implicit exploration of useful
knowledge from each other. During meta-test, binocular embeddings are
aggregated together to support decision-making, which greatly improve the
accuracy of classification. Extensive experiments conducted on multiple
benchmarks including cross-domain validation confirm the effectiveness of our
method.

本研究提出了一种统一框架，Binocular Mutual Learning（BML），通过 intra-view 和 cross-view modeling 实现全局视角和局部视角的兼容性，将双目嵌入聚合在一起支持决策，从而大大提高了分类的准确性。