Differentiable simulators promise faster computation time for reinforcement
learning by replacing zeroth-order gradient estimates of a stochastic objective
with an estimate based on first-order gradients. However, it is yet unclear
what factors decide the performance of the two estimators on complex landscapes
that involve long-horizon planning and control on physical systems, despite the
crucial relevance of this question for the utility of differentiable
simulators. We show that characteristics of certain physical systems, such as
stiffness or discontinuities, may compromise the efficacy of the first-order
estimator, and analyze this phenomenon through the lens of bias and variance.
We additionally propose an $\alpha$-order gradient estimator, with $\alpha \in
[0,1]$, which correctly utilizes exact gradients to combine the efficiency of
first-order estimates with the robustness of zero-order methods. We demonstrate
the pitfalls of traditional estimators and the advantages of the $\alpha$-order
estimator on some numerical examples.

本研究探讨了可微分模拟器在长期规划和物理系统控制等复杂领域中的性能因素，提出了一种 alpha 阶梯度估计器以结合一阶估计的效率和零阶方法的鲁棒性，并在数值示例中证明其优点