Recently, large language models (LLMs) have outperformed human experts in
predicting the results of neuroscience experiments (Luo et al., 2024). What is
the basis for this performance? One possibility is that statistical patterns in
that specific scientific literature, as opposed to emergent reasoning abilities
arising from broader training, underlie LLMs' performance. To evaluate this
possibility, we trained (next word prediction) a relatively small
124M-parameter GPT-2 model on 1.3 billion tokens of domain-specific knowledge.
Despite being orders of magnitude smaller than larger LLMs trained on trillions
of tokens, small models achieved expert-level performance in predicting
neuroscience results. Small models trained on the neuroscience literature
succeeded when they were trained from scratch using a tokenizer specifically
trained on neuroscience text or when the neuroscience literature was used to
finetune a pretrained GPT-2. Our results indicate that expert-level performance
may be attained by even small LLMs through domain-specific, auto-regressive
training approaches.

通过领域特定的自回归训练方法，在小型的 GPT-2 模型上训练神经科学文献，可以实现即使在小规模的语言模型上也能达到专家水平的性能表现。

通过从头开始训练领域知识来匹配领域专家

Matching domain experts by training from scratch on domain knowledge

The emergence of large multimodal models has unlocked remarkable potential in
AI, particularly in pathology. However, the lack of specialized, high-quality
benchmark impeded their development and precise evaluation. To address this, we
introduce PathMMU, the largest and highest-quality expert-validated pathology
benchmark for LMMs. It comprises 33,573 multimodal multi-choice questions and
21,599 images from various sources, and an explanation for the correct answer
accompanies each question. The construction of PathMMU capitalizes on the
robust capabilities of GPT-4V, utilizing approximately 30,000 gathered
image-caption pairs to generate Q\&As. Significantly, to maximize PathMMU's
authority, we invite six pathologists to scrutinize each question under strict
standards in PathMMU's validation and test sets, while simultaneously setting
an expert-level performance benchmark for PathMMU. We conduct extensive
evaluations, including zero-shot assessments of 14 open-sourced and three
closed-sourced LMMs and their robustness to image corruption. We also fine-tune
representative LMMs to assess their adaptability to PathMMU. The empirical
findings indicate that advanced LMMs struggle with the challenging PathMMU
benchmark, with the top-performing LMM, GPT-4V, achieving only a 51.7\%
zero-shot performance, significantly lower than the 71.4\% demonstrated by
human pathologists. After fine-tuning, even open-sourced LMMs can surpass
GPT-4V with a performance of over 60\%, but still fall short of the expertise
shown by pathologists. We hope that the PathMMU will offer valuable insights
and foster the development of more specialized, next-generation LLMs for
pathology.

通过构建专业水准的最大、最高质量的病理学基准测试 PathMMU，我们对大型多模态模型的发展和精确评估进行了研究，发现高级多模态模型在面临挑战性的 PathMMU 基准测试中表现不佳，而且即使进行优化，也无法达到病理学家的专业水准。

PathMMU: 一个用于理解和推理病理学的大规模多模态专业水平基准

PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding  and Reasoning in Pathology

Machine learning algorithms frequently require careful tuning of model
hyperparameters, regularization terms, and optimization parameters.
Unfortunately, this tuning is often a "black art" that requires expert
experience, unwritten rules of thumb, or sometimes brute-force search. Much
more appealing is the idea of developing automatic approaches which can
optimize the performance of a given learning algorithm to the task at hand. In
this work, we consider the automatic tuning problem within the framework of
Bayesian optimization, in which a learning algorithm's generalization
performance is modeled as a sample from a Gaussian process (GP). The tractable
posterior distribution induced by the GP leads to efficient use of the
information gathered by previous experiments, enabling optimal choices about
what parameters to try next. Here we show how the effects of the Gaussian
process prior and the associated inference procedure can have a large impact on
the success or failure of Bayesian optimization. We show that thoughtful
choices can lead to results that exceed expert-level performance in tuning
machine learning algorithms. We also describe new algorithms that take into
account the variable cost (duration) of learning experiments and that can
leverage the presence of multiple cores for parallel experimentation. We show
that these proposed algorithms improve on previous automatic procedures and can
reach or surpass human expert-level optimization on a diverse set of
contemporary algorithms including latent Dirichlet allocation, structured SVMs
and convolutional neural networks.

本研究基于贝叶斯优化框架，建立高斯过程模型，通过前期实验获得的信息来优化机器学习算法的超参数调整，作者提出可以超过经验人类调参表现的自动算法，并介绍收集实验信息、利用多核心并行实验等新算法。