It seems that in the current age, computers, computation, and data have an
increasingly important role to play in scientific research and discovery. This
is reflected in part by the rise of machine learning and artificial
intelligence, which have become great areas of interest not just for computer
science but also for many other fields of study. More generally, there have
been trends moving towards the use of bigger, more complex and higher capacity
models. It also seems that stochastic models, and stochastic variants of
existing deterministic models, have become important research directions in
various fields. For all of these types of models, gradient-based optimization
remains as the dominant paradigm for model fitting, control, and more. This
dissertation considers unconstrained, nonlinear optimization problems, with a
focus on the gradient itself, that key quantity which enables the solution of
such problems.
In chapter 1, we introduce the notion of reverse differentiation, a term
which describes the body of techniques which enables the efficient computation
of gradients. We cover relevant techniques both in the deterministic and
stochastic cases. We present a new framework for calculating the gradient of
problems which involve both deterministic and stochastic elements. In chapter
2, we analyze the properties of the gradient estimator, with a focus on those
properties which are typically assumed in convergence proofs of optimization
algorithms. Chapter 3 gives various examples of applying our new gradient
estimator. We further explore the idea of working with piecewise continuous
models, that is, models with distinct branches and if statements which define
what specific branch to use.

在当今时代，计算机、计算和数据在科学研究和发现中的重要性不断增加。本论文主要关注梯度本身，解决非线性优化问题，并介绍了逆向微分的概念和应用，以及分段连续模型的使用案例。