This paper provides a review and commentary on the past, present, and future
of numerical optimization algorithms in the context of machine learning
applications. Through case studies on text classification and the training of
deep neural networks, we discuss how optimization problems arise in machine
learning and what makes them challenging. A major theme of our study is that
large-scale machine learning represents a distinctive setting in which the
stochastic gradient (SG) method has traditionally played a central role while
conventional gradient-based nonlinear optimization techniques typically falter.
Based on this viewpoint, we present a comprehensive theory of a
straightforward, yet versatile SG algorithm, discuss its practical behavior,
and highlight opportunities for designing algorithms with improved performance.
This leads to a discussion about the next generation of optimization methods
for large-scale machine learning, including an investigation of two main
streams of research on techniques that diminish noise in the stochastic
directions and methods that make use of second-order derivative approximations.

本文讨论了数值优化算法在机器学习应用中的过去、现在和未来。通过文本分类和深度神经网络训练的案例研究，探讨优化问题在机器学习中的出现和挑战，强调了大规模机器学习环境下随机梯度方法的重要性和传统梯度优化方法的局限性。基于这种视角，提出一种简单，通用的随机梯度算法，并讨论其实际表现和改进机会。最后，针对大规模机器学习的下一代优化方法进行了探讨，包括降低随机方向噪声的技术和使用二阶导数近似的方法两大研究方向。

大规模机器学习优化方法

Optimization Methods for Large-Scale Machine Learning

We introduce COCO, an open source platform for Comparing Continuous
Optimizers in a black-box setting. COCO aims at automatizing the tedious and
repetitive task of benchmarking numerical optimization algorithms to the
greatest possible extent. The platform and the underlying methodology allow to
benchmark in the same framework deterministic and stochastic solvers for both
single and multiobjective optimization. We present the rationales behind the
(decade-long) development of the platform as a general proposition for
guidelines towards better benchmarking. We detail underlying fundamental
concepts of COCO such as the definition of a problem as a function instance,
the underlying idea of instances, the use of target values, and runtime defined
by the number of function calls as the central performance measure. Finally, we
give a quick overview of the basic code structure and the currently available
test suites.

COCO 是一个开源平台，旨在自动化比较黑盒优化器。该平台允许使用相同的框架来测试确定性和随机求解器以进行单目标和多目标优化。论文详细介绍了 COCO 的基本概念和代码结构，以及问题和实例的定义、目标值的使用和运行时间的定义等关键概念。

COCO: 在黑盒环境下比较连续优化器的平台

COCO: A Platform for Comparing Continuous Optimizers in a Black-Box  Setting

The implied volatility surface (IVS) is a fundamental building block in
computational finance. We provide a survey of methodologies for constructing
such surfaces. We also discuss various topics which can influence the
successful construction of IVS in practice: arbitrage-free conditions in both
strike and time, how to perform extrapolation outside the core region, choice
of calibrating functional and selection of numerical optimization algorithms,
volatility surface dynamics and asymptotics.

本文综述了构建隐含波动率面的多种方法，同时讨论了多个实际构建过程中的问题，如无套利条件、拟合函数选择和数值优化算法等。