Charts are important for presenting and explaining complex data
relationships. Recently, multimodal large language models (MLLMs) have shown
remarkable capabilities in various chart understanding tasks. However, the
sheer size of these models in terms of parameters and computational
requirements limits their use in resource-constrained environments. In this
paper, we present TinyChart, an efficient MLLM for chart understanding with
only 3B parameters. TinyChart overcomes two key challenges in efficient chart
understanding: (1) reduce the burden of learning numerical computations through
a Program-of-Thoughts (PoT) learning strategy, which trains the model to
generate Python programs for numerical calculations, and (2) reduce lengthy
vision feature sequences produced by the vision transformer for high-resolution
images through a Vision Token Merging module, which gradually merges most
similar vision tokens. Extensive experiments demonstrate that our 3B TinyChart
achieves SOTA performance on a variety of chart understanding benchmarks
including ChartQA, Chart-to-Text, Chart-to-Table, OpenCQA, and ChartX. It
outperforms several chart understanding MLLM with up to 13B parameters such as
ChartLlama and ChartAst, and close-sourced general-purpose MLLM GPT-4V on
ChartQA. It also demonstrates its superior efficiency with higher throughput
during inference due to a smaller model scale and more efficient vision
encoding. Our code and model are available at
this https URL

TinyChart 是一个有效的 MLLM 图表理解框架，仅含 3B 个参数，通过采用思维程序（PoT）学习策略和视觉令牌合并模块，成功解决了图表理解过程中的计算负担和高分辨率图像的视觉特征序列长度问题，并在多个图表理解任务中达到了最先进性能。

TinyChart：使用视觉令牌合并和思维程序学习的高效图表理解

TinyChart: Efficient Chart Understanding with Visual Token Merging and  Program-of-Thoughts Learning

In recent years, the application of multimodal large language models (MLLM)
in various fields has achieved remarkable success. However, as the foundation
model for many downstream tasks, current MLLMs are composed of the well-known
Transformer network, which has a less efficient quadratic computation
complexity. To improve the efficiency of such basic models, we propose Cobra, a
linear computational complexity MLLM. Specifically, Cobra integrates the
efficient Mamba language model into the visual modality. Moreover, we explore
and study various modal fusion schemes to create an effective multi-modal
Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely
competitive performance with current computationally efficient state-of-the-art
methods, \textit{e.g.}, LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster
speed due to Cobra's linear sequential modeling. (2) Interestingly, the results
of closed-set challenging prediction benchmarks show that Cobra performs well
in overcoming visual illusions and spatial relationship judgments. (3) Notably,
Cobra even achieves comparable performance to LLaVA with about 43% of the
number of parameters. We will make all codes of Cobra open-source and hope that
the proposed method can facilitate future research on complexity problems in
MLLM. Our project page is available at: this https URL

通过将高效的 Mamba 语言模型引入视觉模态，Cobra 实现了线性计算复杂度的多模态大型语言模型，其在现有的计算效率高的方法上表现出极具竞争力的性能，并且具有更快的速度；同时，Cobra 在克服视觉错觉和空间关系判断方面表现出色，甚至与 LLaVA 在参数数量只有 43% 时表现相当；我们将开源 Cobra 的所有代码，希望该方法能促进 MLLM 的复杂性问题的未来研究。