Jacobian-Enhanced Neural Networks (JENN) are densely connected multi-layer
perceptrons, whose training process is modified to predict partial derivatives
accurately. Their main benefit is better accuracy with fewer training points
compared to standard neural networks. These attributes are particularly
desirable in the field of computer-aided design, where there is often the need
to replace computationally expensive, physics-based models with fast running
approximations, known as surrogate models or meta-models. Since a surrogate
emulates the original model accurately in near-real time, it yields a speed
benefit that can be used to carry out orders of magnitude more function calls
quickly. However, in the special case of gradient-enhanced methods, there is
the additional value proposition that partial derivatives are accurate, which
is a critical property for one important use-case: surrogate-based
optimization. This work derives the complete theory and exemplifies its
superiority over standard neural nets for surrogate-based optimization.

Jacobian-Enhanced Neural Networks (JENN) are used in computer-aided design to accurately predict partial derivatives and provide a faster alternative to physics-based models known as surrogate models or meta-models, particularly for surrogate-based optimization.

雅可比增强神经网络

Jacobian-Enhanced Neural Networks

We introduce a surrogate-based black-box optimization method, termed
Polynomial-model-based optimization (PMBO). The algorithm alternates polynomial
approximation with Bayesian optimization steps, using Gaussian processes to
model the error between the objective and its polynomial fit. We describe the
algorithmic design of PMBO and compare the results of the performance of PMBO
with several optimization methods for a set of analytic test functions.
The results show that PMBO outperforms the classic Bayesian optimization and
is robust with respect to the choice of its correlation function family and its
hyper-parameter setting, which, on the contrary, need to be carefully tuned in
classic Bayesian optimization. Remarkably, PMBO performs comparably with
state-of-the-art evolutionary algorithms such as the Covariance Matrix
Adaptation -- Evolution Strategy (CMA-ES). This finding suggests that PMBO
emerges as the pivotal choice among surrogate-based optimization methods when
addressing low-dimensional optimization problems. Hereby, the simple nature of
polynomials opens the opportunity for interpretation and analysis of the
inferred surrogate model, providing a macroscopic perspective on the landscape
of the objective function.

采用多项式模型为基础的优化方法 (PMBO) 在低维优化问题中表现出色，与经典贝叶斯优化和进化算法相比具有更好的鲁棒性和解释性。

PMBO: 通过多元多项式替代物强化黑箱优化

PMBO: Enhancing Black-Box Optimization through Multivariate Polynomial  Surrogates

Hyperparameter optimization is the process of identifying the appropriate
hyperparameter configuration of a given machine learning model with regard to a
given learning task. For smaller data sets, an exhaustive search is possible;
However, when the data size and model complexity increase, the number of
configuration evaluations becomes the main computational bottleneck. A
promising paradigm for tackling this type of problem is surrogate-based
optimization. The main idea underlying this paradigm considers an incrementally
updated model of the relation between the hyperparameter space and the output
(target) space; the data for this model are obtained by evaluating the main
learning engine, which is, for example, a factorization machine-based model. By
learning to approximate the hyperparameter-target relation, the surrogate
(machine learning) model can be used to score large amounts of hyperparameter
configurations, exploring parts of the configuration space beyond the reach of
direct machine learning engine evaluation. Commonly, a surrogate is selected
prior to optimization initialization and remains the same during the search. We
investigated whether dynamic switching of surrogates during the optimization
itself is a sensible idea of practical relevance for selecting the most
appropriate factorization machine-based models for large-scale online
recommendation. We conducted benchmarks on data sets containing hundreds of
millions of instances against established baselines such as Random Forest- and
Gaussian process-based surrogates. The results indicate that surrogate
switching can offer good performance while considering fewer learning engine
evaluations.

研究表明，在大规模在线推荐系统中，采用基于代理模型的最优化算法可以通过动态切换代理模型而对海量的超参数配置进行评估来提高模型指标。

动态代理切换：在线推荐中因子机配置的高效采样搜索

Dynamic Surrogate Switching: Sample-Efficient Search for Factorization Machine Configurations in Online Recommendations

This paper reports on the first international competition on AI for the
traveling salesman problem (TSP) at the International Joint Conference on
Artificial Intelligence 2021 (IJCAI-21). The TSP is one of the classical
combinatorial optimization problems, with many variants inspired by real-world
applications. This first competition asked the participants to develop
algorithms to solve a time-dependent orienteering problem with stochastic
weights and time windows (TD-OPSWTW). It focused on two types of learning
approaches: surrogate-based optimization and deep reinforcement learning. In
this paper, we describe the problem, the setup of the competition, the winning
methods, and give an overview of the results. The winning methods described in
this work have advanced the state-of-the-art in using AI for stochastic routing
problems. Overall, by organizing this competition we have introduced routing
problems as an interesting problem setting for AI researchers. The simulator of
the problem has been made open-source and can be used by other researchers as a
benchmark for new AI methods.

介绍 2021 年国际人工智能联合大会（IJCAI-21）上第一个旅行推销员问题（TSP）人工智能竞赛，关注基于学习的算法，介绍了该竞赛的问题、比赛安排、获胜方法和结果概述。竞赛的胜利方法使 AI 在随机路由问题上取得了技术进步，也使路由问题成为 AI 研究人员感兴趣的问题设定。同时，该比赛所使用的路由问题模拟器现已成为开源工具，可以成为其他研究人员使用的新 AI 方法基准测试。