We introduce a new challenge to test the STEM skills of neural models. The
problems in the real world often require solutions, combining knowledge from
STEM (science, technology, engineering, and math). Unlike existing datasets,
our dataset requires the understanding of multimodal vision-language
information of STEM. Our dataset features one of the largest and most
comprehensive datasets for the challenge. It includes 448 skills and 1,073,146
questions spanning all STEM subjects. Compared to existing datasets that often
focus on examining expert-level ability, our dataset includes fundamental
skills and questions designed based on the K-12 curriculum. We also add
state-of-the-art foundation models such as CLIP and GPT-3.5-Turbo to our
benchmark. Results show that the recent model advances only help master a very
limited number of lower grade-level skills (2.5% in the third grade) in our
dataset. In fact, these models are still well below (averaging 54.7%) the
performance of elementary students, not to mention near expert-level
performance. To understand and increase the performance on our dataset, we
teach the models on a training split of our dataset. Even though we observe
improved performance, the model performance remains relatively low compared to
average elementary students. To solve STEM problems, we will need novel
algorithmic innovations from the community.

我们介绍了一个新的挑战来测试神经模型的 STEM 技能，我们的数据集涵盖了 STEM 的多模式视觉语言信息，包括了 448 项技能和 1,073,146 个问题，与现有数据集相比，我们的数据集涵盖了从幼儿园到 12 年级课程的基础技能和问题，并添加了 CLIP 和 GPT-3.5-Turbo 等最新的基础模型到我们的基准测试，结果表明，最近的模型进展只有在我们数据集中的一小部分低年级技能（三年级的 2.5%）上有所帮助，事实上，这些模型的性能仍然远低于小学生的平均水平（仅平均 54.7%），更不用说接近专家级性能了，为了提高我们数据集上的模型性能，我们将模型训练在数据集的训练集上，尽管我们观察到性能有所提升，但与普通小学生相比，模型性能仍然相对较低，因此我们需要来自社区的创新算法来解决 STEM 问题。

测量神经模型的视觉 - 语言 STEM 技能

Measuring Vision-Language STEM Skills of Neural Models

AI research is increasingly industry-driven, making it crucial to understand
company contributions to this field. We compare leading AI companies by
research publications, citations, size of training runs, and contributions to
algorithmic innovations. Our analysis reveals the substantial role played by
Google, OpenAI and Meta. We find that these three companies have been
responsible for some of the largest training runs, developed a large fraction
of the algorithmic innovations that underpin large language models, and led in
various metrics of citation impact. In contrast, leading Chinese companies such
as Tencent and Baidu had a lower impact on many of these metrics compared to US
counterparts. We observe many industry labs are pursuing large training runs,
and that training runs from relative newcomers -- such as OpenAI and Anthropic
-- have matched or surpassed those of long-standing incumbents such as Google.
The data reveals a diverse ecosystem of companies steering AI progress, though
US labs such as Google, OpenAI and Meta lead across critical metrics.

AI 研究越来越多地由行业推动，因此了解企业对该领域的贡献至关重要。我们通过研究发表论文、引用、训练规模和算法创新来比较领先的 AI 公司。我们的分析揭示了 Google、OpenAI 和 Meta 的重大作用。这三家公司负责一些最大规模的训练，发展了支撑大型语言模型的大部分算法创新，并在引用影响力等各项指标中处于领先地位。相比之下，腾讯和百度等国内领先公司在许多指标上的影响力较低。我们观察到许多行业实验室正在进行大规模训练，而 OpenAI 和 Anthropic 等新进公司的训练规模甚至超过了 Google 等历史悠久的老牌公司。数据显示了一个多元化的公司生态系统在推动 AI 进展，尽管美国的实验室如 Google、OpenAI 和 Meta 在关键指标上处于领导地位。

谁在引领人工智能？行业 AI 研究分析

Who is leading in AI? An analysis of industry AI research

Machine learning has achieved remarkable success over the past couple of
decades, often attributed to a combination of algorithmic innovations and the
availability of high-quality data available at scale. However, a third critical
component is the fine-tuning of hyperparameters, which plays a pivotal role in
achieving optimal model performance. Despite its significance, hyperparameter
optimization (HPO) remains a challenging task for several reasons. Many HPO
techniques rely on naive search methods or assume that the loss function is
smooth and continuous, which may not always be the case. Traditional methods,
like grid search and Bayesian optimization, often struggle to quickly adapt and
efficiently search the loss landscape. Grid search is computationally
expensive, while Bayesian optimization can be slow to prime. Since the search
space for HPO is frequently high-dimensional and non-convex, it is often
challenging to efficiently find a global minimum. Moreover, optimal
hyperparameters can be sensitive to the specific dataset or task, further
complicating the search process. To address these issues, we propose a new
hyperparameter optimization method, HomOpt, using a data-driven approach based
on a generalized additive model (GAM) surrogate combined with homotopy
optimization. This strategy augments established optimization methodologies to
boost the performance and effectiveness of any given method with faster
convergence to the optimum on continuous, discrete, and categorical domain
spaces. We compare the effectiveness of HomOpt applied to multiple optimization
techniques (e.g., Random Search, TPE, Bayes, and SMAC) showing improved
objective performance on many standardized machine learning benchmarks and
challenging open-set recognition tasks.

使用基于广义加法模型（GAM）替代与同伦优化结合的数据驱动方法，我们提出了一种新的超参数优化方法 HomOpt，以提高现有方法在连续、离散和分类域空间上的性能和效果，并在多个优化技术（如随机搜索、TPE、贝叶斯和 SMAC）应用 HomOpt，展示了对许多标准机器学习基准和具有挑战性的开集识别任务的改进性能。