Recent advancements in language models have showcased human-comparable
performance in academic entrance exams. However, existing studies often
overlook questions that require the integration of visual comprehension, thus
compromising the full spectrum and complexity inherent in real-world scenarios.
To address this gap, we present a comprehensive framework to evaluate language
models on entrance exams, which incorporates both textual and visual elements.
We evaluate the two most recent editions of Exame Nacional do Ensino M\'edio
(ENEM), the main standardized entrance examination adopted by Brazilian
universities. Our study not only reaffirms the capabilities of GPT-4 as the
state of the art for handling complex multidisciplinary questions, but also
pioneers in offering a realistic assessment of multimodal language models on
Portuguese examinations. One of the highlights is that text captions
transcribing visual content outperform the direct use of images, suggesting
that the vision model has room for improvement. Yet, despite improvements
afforded by images or captions, mathematical questions remain a challenge for
these state-of-the-art models. The code and data used on experiments are
available at this https URL

通过综合文本和视觉元素，对语言模型在入学考试上进行评估的综合框架表明 GPT-4 在处理复杂多学科问题方面具有最先进的能力，但数学问题仍然是这些模型的挑战。

评估 GPT-4 在巴西大学入学考试中的视觉能力

Evaluating GPT-4's Vision Capabilities on Brazilian University Admission  Exams

Understanding mathematical questions effectively is a crucial task, which can
benefit many applications, such as difficulty estimation. Researchers have
drawn much attention to designing pre-training models for question
representations due to the scarcity of human annotations (e.g., labeling
difficulty). However, unlike general free-format texts (e.g., user comments),
mathematical questions are generally designed with explicit purposes and
mathematical logic, and usually consist of more complex content, such as
formulas, and related mathematical knowledge (e.g., Function). Therefore, the
problem of holistically representing mathematical questions remains
underexplored. To this end, in this paper, we propose a novel contrastive
pre-training approach for mathematical question representations, namely QuesCo,
which attempts to bring questions with more similar purposes closer.
Specifically, we first design two-level question augmentations, including
content-level and structure-level, which generate literally diverse question
pairs with similar purposes. Then, to fully exploit hierarchical information of
knowledge concepts, we propose a knowledge hierarchy-aware rank strategy
(KHAR), which ranks the similarities between questions in a fine-grained
manner. Next, we adopt a ranking contrastive learning task to optimize our
model based on the augmented and ranked questions. We conduct extensive
experiments on two real-world mathematical datasets. The experimental results
demonstrate the effectiveness of our model.

本研究提出了一种新的数学问题对比预训练方法 QuesCo，该方法利用知识层次结构感知的排名策略来更好地表示数学问题，并进行了广泛的实验以证明其有效性。