Deep reinforcement learning can generate complex control policies, but
requires large amounts of training data to work effectively. Recent work has
attempted to address this issue by leveraging differentiable simulators.
However, inherent problems such as local minima and exploding/vanishing
numerical gradients prevent these methods from being generally applied to
control tasks with complex contact-rich dynamics, such as humanoid locomotion
in classical RL benchmarks. In this work we present a high-performance
differentiable simulator and a new policy learning algorithm (SHAC) that can
effectively leverage simulation gradients, even in the presence of
non-smoothness. Our learning algorithm alleviates problems with local minima
through a smooth critic function, avoids vanishing/exploding gradients through
a truncated learning window, and allows many physical environments to be run in
parallel. We evaluate our method on classical RL control tasks, and show
substantial improvements in sample efficiency and wall-clock time over
state-of-the-art RL and differentiable simulation-based algorithms. In
addition, we demonstrate the scalability of our method by applying it to the
challenging high-dimensional problem of muscle-actuated locomotion with a large
action space, achieving a greater than 17x reduction in training time over the
best-performing established RL algorithm.

本文提出 SHAC 算法，该算法基于高性能可微分模拟器，可以有效利用模拟梯度，避免局部最小值问题，并通过截断学习窗口来避免梯度消失或爆炸。在控制任务中表现出更高的样本效率和更短的训练时间。

带有并行可微模拟的加速策略学习

Accelerated Policy Learning with Parallel Differentiable Simulation

Differentiable simulators promise faster computation time for reinforcement
learning by replacing zeroth-order gradient estimates of a stochastic objective
with an estimate based on first-order gradients. However, it is yet unclear
what factors decide the performance of the two estimators on complex landscapes
that involve long-horizon planning and control on physical systems, despite the
crucial relevance of this question for the utility of differentiable
simulators. We show that characteristics of certain physical systems, such as
stiffness or discontinuities, may compromise the efficacy of the first-order
estimator, and analyze this phenomenon through the lens of bias and variance.
We additionally propose an $\alpha$-order gradient estimator, with $\alpha \in
[0,1]$, which correctly utilizes exact gradients to combine the efficiency of
first-order estimates with the robustness of zero-order methods. We demonstrate
the pitfalls of traditional estimators and the advantages of the $\alpha$-order
estimator on some numerical examples.

本研究探讨了可微分模拟器在长期规划和物理系统控制等复杂领域中的性能因素，提出了一种 alpha 阶梯度估计器以结合一阶估计的效率和零阶方法的鲁棒性，并在数值示例中证明其优点

可微模拟器是否提供更好的策略梯度？

Do Differentiable Simulators Give Better Policy Gradients?

Differentiable simulators provide an avenue for closing the sim-to-real gap
by enabling the use of efficient, gradient-based optimization algorithms to
find the simulation parameters that best fit the observed sensor readings.
Nonetheless, these analytical models can only predict the dynamical behavior of
systems for which they have been designed. In this work, we study the
augmentation of a novel differentiable rigid-body physics engine via neural
networks that is able to learn nonlinear relationships between dynamic
quantities and can thus learn effects not accounted for in traditional
simulators.Such augmentations require less data to train and generalize better
compared to entirely data-driven models. Through extensive experiments, we
demonstrate the ability of our hybrid simulator to learn complex dynamics
involving frictional contacts from real data, as well as match known models of
viscous friction, and present an approach for automatically discovering useful
augmentations. We show that, besides benefiting dynamics modeling, inserting
neural networks can accelerate model-based control architectures. We observe a
ten-fold speed-up when replacing the QP solver inside a model-predictive gait
controller for quadruped robots with a neural network, allowing us to
significantly improve control delays as we demonstrate in real-hardware
experiments.
We publish code, additional results and videos from our experiments on our
project webpage at this https URL

本文介绍了一种基于神经网络的不同 iable rigid-body 物理引擎来学习动力学关系以及对其进行自动发现，替代了常规的解析式建模，使用神经网络能加速基于模型的控制架构，提高建模和控制的准确度。

NeuralSim：用神经网络增强可微分模拟器

NeuralSim: Augmenting Differentiable Simulators with Neural Networks

Physical simulators have been widely used in robot planning and control.
Among them, differentiable simulators are particularly favored, as they can be
incorporated into gradient-based optimization algorithms that are efficient in
solving inverse problems such as optimal control and motion planning.
Simulating deformable objects is, however, more challenging compared to rigid
body dynamics. The underlying physical laws of deformable objects are more
complex, and the resulting systems have orders of magnitude more degrees of
freedom and therefore they are significantly more computationally expensive to
simulate. Computing gradients with respect to physical design or controller
parameters is typically even more computationally challenging. In this paper,
we propose a real-time, differentiable hybrid Lagrangian-Eulerian physical
simulator for deformable objects, ChainQueen, based on the Moving Least Squares
Material Point Method (MLS-MPM). MLS-MPM can simulate deformable objects
including contact and can be seamlessly incorporated into inference, control
and co-design systems. We demonstrate that our simulator achieves high
precision in both forward simulation and backward gradient computation. We have
successfully employed it in a diverse set of control tasks for soft robots,
including problems with nearly 3,000 decision variables.

本文介绍了一种针对可形变物体的实时可分化杂交拉格朗日 - 欧拉物理仿真器 ChainQueen，基于 MLS-MPM，能够高效地模拟包括接触在内的形变物体，并在控制任务中取得较高的精度和性能。