Charts are important for presenting and explaining complex data
relationships. Recently, multimodal large language models (MLLMs) have shown
remarkable capabilities in various chart understanding tasks. However, the
sheer size of these models in terms of parameters and computational
requirements limits their use in resource-constrained environments. In this
paper, we present TinyChart, an efficient MLLM for chart understanding with
only 3B parameters. TinyChart overcomes two key challenges in efficient chart
understanding: (1) reduce the burden of learning numerical computations through
a Program-of-Thoughts (PoT) learning strategy, which trains the model to
generate Python programs for numerical calculations, and (2) reduce lengthy
vision feature sequences produced by the vision transformer for high-resolution
images through a Vision Token Merging module, which gradually merges most
similar vision tokens. Extensive experiments demonstrate that our 3B TinyChart
achieves SOTA performance on a variety of chart understanding benchmarks
including ChartQA, Chart-to-Text, Chart-to-Table, OpenCQA, and ChartX. It
outperforms several chart understanding MLLM with up to 13B parameters such as
ChartLlama and ChartAst, and close-sourced general-purpose MLLM GPT-4V on
ChartQA. It also demonstrates its superior efficiency with higher throughput
during inference due to a smaller model scale and more efficient vision
encoding. Our code and model are available at
this https URL

TinyChart 是一个有效的 MLLM 图表理解框架，仅含 3B 个参数，通过采用思维程序（PoT）学习策略和视觉令牌合并模块，成功解决了图表理解过程中的计算负担和高分辨率图像的视觉特征序列长度问题，并在多个图表理解任务中达到了最先进性能。