The automatic design of robots has existed for 30 years but has been
constricted by serial non-differentiable design evaluations, premature
convergence to simple bodies or clumsy behaviors, and a lack of sim2real
transfer to physical machines. Thus, here we employ massively-parallel
differentiable simulations to rapidly and simultaneously optimize individual
neural control of behavior across a large population of candidate body plans
and return a fitness score for each design based on the performance of its
fully optimized behavior. Non-differentiable changes to the mechanical
structure of each robot in the population -- mutations that rearrange, combine,
add, or remove body parts -- were applied by a genetic algorithm in an outer
loop of search, generating a continuous flow of novel morphologies with
highly-coordinated and graceful behaviors honed by gradient descent. This
enabled the exploration of several orders-of-magnitude more designs than all
previous methods, despite the fact that robots here have the potential to be
much more complex, in terms of number of independent motors, than those in
prior studies. We found that evolution reliably produces ``increasingly
differentiable'' robots: body plans that smooth the loss landscape in which
learning operates and thereby provide better training paths toward performant
behaviors. Finally, one of the highly differentiable morphologies discovered in
simulation was realized as a physical robot and shown to retain its optimized
behavior. This provides a cyberphysical platform to investigate the
relationship between evolution and learning in biological systems and broadens
our understanding of how a robot's physical structure can influence the ability
to train policies for it. Videos and code at
this https URL

通过大规模并行可微分仿真，利用遗传算法对机器人进行自动设计，通过优化机器人的行为并探索不同机械结构的变化，发现进化可产生逐渐可微分的机器人，在实体机器人上验证了优化行为的可行性，从而拓宽了我们对生物系统中进化和学习之间关系以及机器人物理结构对策略训练能力影响的理解。

可微分机器人的演化与学习

Evolution and learning in differentiable robots

Learning algorithms, like Quality-Diversity (QD), can be used to acquire
repertoires of diverse robotics skills. This learning is commonly done via
computer simulation due to the large number of evaluations required. However,
training in a virtual environment generates a gap between simulation and
reality. Here, we build upon the Reset-Free QD (RF-QD) algorithm to learn
controllers directly on a physical robot. This method uses a dynamics model,
learned from interactions between the robot and the environment, to predict the
robot's behaviour and improve sample efficiency. A behaviour selection policy
filters out uninteresting or unsafe policies predicted by the model. RF-QD also
includes a recovery policy that returns the robot to a safe zone when it has
walked outside of it, allowing continuous learning. We demonstrate that our
method enables a physical quadruped robot to learn a repertoire of behaviours
in two hours without human supervision. We successfully test the solution
repertoire using a maze navigation task. Finally, we compare our approach to
the MAP-Elites algorithm. We show that dynamics awareness and a recovery policy
are required for training on a physical robot for optimal archive generation.
Video available at this https URL

通过 Reset-Free QD 算法和动力学模型，在物理机器人上直接学习控制器，以提高样本效率和生成最佳存档，从而使物理四足机器人在两小时内学习到行为技能库。