An oft-ignored challenge of real-world reinforcement learning is that the
real world does not pause when agents make learning updates. As standard
simulated environments do not address this real-time aspect of learning, most
available implementations of RL algorithms process environment interactions and
learning updates sequentially. As a consequence, when such implementations are
deployed in the real world, they may make decisions based on significantly
delayed observations and not act responsively. Asynchronous learning has been
proposed to solve this issue, but no systematic comparison between sequential
and asynchronous reinforcement learning was conducted using real-world
environments. In this work, we set up two vision-based tasks with a robotic
arm, implement an asynchronous learning system that extends a previous
architecture, and compare sequential and asynchronous reinforcement learning
across different action cycle times, sensory data dimensions, and mini-batch
sizes. Our experiments show that when the time cost of learning updates
increases, the action cycle time in sequential implementation could grow
excessively long, while the asynchronous implementation can always maintain an
appropriate action cycle time. Consequently, when learning updates are
expensive, the performance of sequential learning diminishes and is
outperformed by asynchronous learning by a substantial margin. Our system
learns in real-time to reach and track visual targets from pixels within two
hours of experience and does so directly using real robots, learning completely
from scratch.

本文论述了异步学习和顺序学习的比较，并在真实环境下使用机器人手臂和视觉任务进行了实验。研究结果表明，当学习更新的时间成本增加时，顺序学习的性能会显著下降，而异步学习会明显胜过顺序学习。

物理机器人实时控制的异步强化学习

Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots

Human-Object Interaction (HOI) detection devotes to learn how humans interact
with surrounding objects via inferring triplets of < human, verb, object >.
However, recent HOI detection methods mostly rely on additional annotations
(e.g., human pose) and neglect powerful interactive reasoning beyond
convolutions. In this paper, we present a novel graph-based interactive
reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs, in
which interactive semantics implied among visual targets are efficiently
exploited. The proposed model consists of a project function that maps related
targets from convolution space to a graph-based semantic space, a message
passing process propagating semantics among all nodes and an update function
transforming the reasoned nodes back to convolution space. Furthermore, we
construct a new framework to assemble in-Graph models for detecting HOIs,
namely in-GraphNet. Beyond inferring HOIs using instance features respectively,
the framework dynamically parses pairwise interactive semantics among visual
targets by integrating two-level in-Graphs, i.e., scene-wide and instance-wide
in-Graphs. Our framework is end-to-end trainable and free from costly
annotations like human pose. Extensive experiments show that our proposed
framework outperforms existing HOI detection methods on both V-COCO and
HICO-DET benchmarks and improves the baseline about 9.4% and 15% relatively,
validating its efficacy in detecting HOIs.

本文提出了一种名为交互图的图形交互推理模型，以推断人类和周围物体的相互作用，并构建了新的框架用于检测 HOIs，即 in-GraphNet，该模型能够有效地利用视觉目标间的交互语义，且不需要昂贵的注释，实验证明该方法在 V-COCO 和 HICO-DET 基准测试中比现有的 HOI 检测方法表现更优，基准线相对提高了 9.4% 和 15%。