Visual language data such as plots, charts, and infographics are ubiquitous
in the human world. However, state-of-the-art vision-language models do not
perform well on these data. We propose MatCha (Math reasoning and Chart
derendering pretraining) to enhance visual language models' capabilities in
jointly modeling charts/plots and language data. Specifically, we propose
several pretraining tasks that cover plot deconstruction and numerical
reasoning which are the key capabilities in visual language modeling.
We perform the MatCha pretraining starting from Pix2Struct, a recently
proposed image-to-text visual language model. On standard benchmarks such as
PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as
much as nearly 20%. We also examine how well MatCha pretraining transfers to
domains such as screenshots, textbook diagrams, and document figures and
observe overall improvement, verifying the usefulness of MatCha pretraining on
broader visual language tasks.

MatCha 通过数学推理与图解构造预训练，增强了视觉语言模型的建模能力，成功提高了如 PlotQA 和 ChartQA 等标准基准测试的表现，证实了该方法在更广泛的视觉语言任务上的有效性和可行性。