In the past year, the capabilities of Multimodal Large Language Models (MLLMs) have significantly improved across various aspects. However, constrained by the quadratic growth of computation in LLMs as the number of tokens increases, efficiency has become a bottleneck for further scaling MLLMs. Although recent efforts have been made to prune visual tokens or use more lightweight LLMs to reduce computation, the problem of quadratic growth in computation with the increase of visual tokens still persists. To address this, we propose a novel approach: instead of reducing the input visual tokens for LLMs, we focus on pruning vision-related computations within the LLMs. After pruning, the computation growth in the LLM is no longer quadratic with the increase of visual tokens, but linear. Surprisingly, we found that after applying such extensive pruning, the capabilities of MLLMs are comparable with the original one and even superior on some benchmarks with only 25% of the computation. This finding opens up the possibility for MLLMs to incorporate much denser visual tokens. Additionally, based on this finding, we further analyzed some architectural design deficiencies in existing MLLMs and proposed promising improvements. To the best of our knowledge, this is the first study to investigate the computational redundancy in the LLM's vision component of MLLMs. Code and checkpoints will be released soon.

本研究解决了多模态大型语言模型（MLLMs）计算效率瓶颈问题，特别是在视觉令牌数量增加导致的二次增长计算问题。研究提出通过剪枝视觉相关计算而非输入降低视觉令牌，这一新方法实现了计算增长由二次转为线性，且经过剪枝后，模型能力在某些基准测试上已超越原模型，仅需25%的计算量。这一发现开启了MLLMs引入更密集视觉令牌的可能性。

二次方程并非多模态大型语言模型所需