With the rapid development of Large Language Models (LLMs), it is crucial to
have benchmarks which can evaluate the ability of LLMs on different domains.
One common use of LLMs is performing tasks on scientific topics, such as
writing algorithms, querying databases or giving mathematical proofs. Inspired
by the way university students are evaluated on such tasks, in this paper, we
propose SciEx - a benchmark consisting of university computer science exam
questions, to evaluate LLMs ability on solving scientific tasks. SciEx is (1)
multilingual, containing both English and German exams, and (2) multi-modal,
containing questions that involve images, and (3) contains various types of
freeform questions with different difficulty levels, due to the nature of
university exams. We evaluate the performance of various state-of-the-art LLMs
on our new benchmark. Since SciEx questions are freeform, it is not
straightforward to evaluate LLM performance. Therefore, we provide human expert
grading of the LLM outputs on SciEx. We show that the free-form exams in SciEx
remain challenging for the current LLMs, where the best LLM only achieves
59.4\% exam grade on average. We also provide detailed comparisons between LLM
performance and student performance on SciEx. To enable future evaluation of
new LLMs, we propose using LLM-as-a-judge to grade the LLM answers on SciEx.
Our experiments show that, although they do not perform perfectly on solving
the exams, LLMs are decent as graders, achieving 0.948 Pearson correlation with
expert grading.

大型语言模型的基准评估 SciEx 提出，用于评估 LLM 在解决科学任务上的能力，并通过比较 LLM 和学生在 SciEx 上的表现，提出了 LLM 作为评分者的方法。

SciEx：基于人工评分和自动评分的科学考试大型语言模型评测

SciEx: Benchmarking Large Language Models on Scientific Exams with Human  Expert Grading and Automatic Grading

Large language models record impressive performance on many natural language
processing tasks. However, their knowledge capacity is limited to the
pretraining corpus. Retrieval augmentation offers an effective solution by
retrieving context from external knowledge sources to complement the language
model. However, existing retrieval augmentation techniques ignore the
structural relationships between these documents. Furthermore, retrieval models
are not explored much in scientific tasks, especially in regard to the
faithfulness of retrieved documents. In this paper, we propose a novel
structure-aware retrieval augmented language model that accommodates document
structure during retrieval augmentation. We create a heterogeneous document
graph capturing multiple types of relationships (e.g., citation, co-authorship,
etc.) that connect documents from more than 15 scientific disciplines (e.g.,
Physics, Medicine, Chemistry, etc.). We train a graph neural network on the
curated document graph to act as a structural encoder for the corresponding
passages retrieved during the model pretraining. Particularly, along with text
embeddings of the retrieved passages, we obtain structural embeddings of the
documents (passages) and fuse them together before feeding them to the language
model. We evaluate our model extensively on various scientific benchmarks that
include science question-answering and scientific document classification
tasks. Experimental results demonstrate that structure-aware retrieval improves
retrieving more coherent, faithful and contextually relevant passages, while
showing a comparable performance in the overall accuracy.

我们提出了一种结构感知的检索增强语言模型，通过在预训练模型中检索并融合具有结构关系的科学文档，来改善检索到的内容的相关性、准确性和连贯性。

ATLANTIC: 结构感知的检索增强语言模型用于跨学科科学

ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for  Interdisciplinary Science

Inspired by recent and revolutionary developments in AI, particularly in
language understanding and generation, we set about designing AI systems that
are able to address complex scientific tasks that challenge human capabilities
to make new discoveries. Central to our approach is the notion of natural
language as core representation, reasoning, and exchange format between
scientific AI and human scientists. In this paper, we identify and discuss some
of the main research challenges to accomplish such vision.

基于 AI 在自然语言理解和生成方面的革命性进展，我们设计了能够解决复杂科学任务的 AI 系统，并将自然语言作为 AI 与人类科学家之间的核心表示、推理和交换格式。本文旨在探讨实现这一愿景所面临的主要研究挑战。

面向语言驱动的科学人工智能

Towards Language-driven Scientific AI

Transformer-based masked language models trained on general corpora, such as
BERT and RoBERTa, have shown impressive performance on various downstream
tasks. Increasingly, researchers are "finetuning" these models to improve
performance on domain-specific tasks. Here, we report a broad study in which we
applied 14 transformer-based models to 11 scientific tasks in order to evaluate
how downstream performance is affected by changes along various dimensions
(e.g., training data, model size, pretraining time, finetuning length). In this
process, we created the largest and most diverse scientific language model to
date, ScholarBERT, by training a 770M-parameter BERT model on an 221B token
scientific literature dataset spanning many disciplines. Counterintuitively,
our evaluation of the 14 BERT-based models (seven versions of ScholarBERT, five
science-specific large language models from the literature, BERT-Base, and
BERT-Large) reveals little difference in performance across the 11
science-focused tasks, despite major differences in model size and training
data. We argue that our results establish an upper bound for the performance
achievable with BERT-based architectures on tasks from the scientific domain.

本文研究了 14 个基于 transformer 的模型在 11 个科学任务上的表现，并创建了迄今最大、最多样化的科学语言模型 ScholarBERT。然而，在模型大小和训练数据等方面存在重大差异的情况下，我们发现这些模型在这些科学任务上的表现几乎没有差异，这一结果为基于 BERT 架构在科学领域任务中的表现建立了一个上限。