We are interested in assessing the use of neural networks as surrogate models to approximate and minimize objective functions in optimization problems. While neural networks are widely used for machine learning tasks such as classification and regression, their application in solving optimization problems has been limited. Our study begins by determining the best activation function for approximating the objective functions of popular nonlinear optimization test problems, and the evidence provided shows that~SiLU has the best performance. We then analyze the accuracy of function value, gradient, and Hessian approximations for such objective functions obtained through interpolation/regression models and neural networks. When compared to interpolation/regression models, neural networks can deliver competitive zero- and first-order approximations (at a high training cost) but underperform on second-order approximation. However, it is shown that combining a neural net activation function with the natural basis for quadratic interpolation/regression can waive the necessity of including cross terms in the natural basis, leading to models with fewer parameters to determine. Lastly, we provide evidence that the performance of a state-of-the-art derivative-free optimization algorithm can hardly be improved when the gradient of an objective function is approximated using any of the surrogate models considered, including neural networks.

我们研究了神经网络作为替代模型来近似和最小化优化问题中的目标函数的使用，通过确定适合目前非线性优化测试问题目标函数近似的最佳激活函数来提供证明，我们分析通过插值/回归模型和神经网络获得的函数值、梯度和Hessian矩阵的近似精度，结果显示神经网络在零阶和一阶近似方面表现出较高竞争力（对应较高的训练成本），但在二阶近似方面表现较差。然而，通过将神经网络激活函数与二次插值/回归的自然基组合，可以减少模型参数数量。最后，我们提供了证据表明，包括神经网络在内的任何考虑的替代模型用于逼近优化算法的梯度时，都无法明显改善目前最先进的无导数优化算法的性能。

神经网络在近似和优化方面的限制