Though robot learning is often formulated in terms of discrete-time Markov
decision processes (MDPs), physical robots require near-continuous multiscale
feedback control. Machines operate on multiple asynchronous sensing modalities,
each with different frequencies, e.g., video frames at 30Hz, proprioceptive
state at 100Hz, force-torque data at 500Hz, etc. While the classic approach is
to batch observations into fixed-time windows then pass them through
feed-forward encoders (e.g., with deep networks), we show that there exists a
more elegant approach -- one that treats policy learning as modeling latent
state dynamics in continuous-time. Specifically, we present 'InFuser', a
unified architecture that trains continuous time-policies with Neural
Controlled Differential Equations (CDEs). InFuser evolves a single latent state
representation over time by (In)tegrating and (Fus)ing multi-sensory
observations (arriving at different frequencies), and inferring actions in
continuous-time. This enables policies that can react to multi-frequency multi
sensory feedback for truly end-to-end visuomotor control, without discrete-time
assumptions. Behavior cloning experiments demonstrate that InFuser learns
robust policies for dynamic tasks (e.g., swinging a ball into a cup) notably
outperforming several baselines in settings where observations from one sensing
modality can arrive at much sparser intervals than others.

本文提出了一种新的方法 InFuser 来训练连续时间策略，通过积分和融合多传感器观察数据来推断出连续时间动作，以期获得更好的视觉 - 运动控制。行为克隆实验表明，InFuser 在动态任务中（例如将球挥入杯中）学习到了稳健的策略，并且在观察数据不均匀的情况下优于一些基线。