Modeling instance-level context and object-object relationships is extremely
challenging. It requires reasoning about bounding boxes of different classes,
locations \etc. Above all, instance-level spatial reasoning inherently requires
modeling conditional distributions on previous detections. Unfortunately, our
current object detection systems do not have any {\bf memory} to remember what
to condition on! The state-of-the-art object detectors still detect all object
in parallel followed by non-maximal suppression (NMS). While memory has been
used for tasks such as captioning, they mostly use image-level memory cells
without capturing the spatial layout. On the other hand, modeling object-object
relationships requires {\bf spatial} reasoning -- not only do we need a memory
to store the spatial layout, but also a effective reasoning module to extract
spatial patterns. This paper presents a conceptually simple yet powerful
solution -- Spatial Memory Network (SMN), to model the instance-level context
efficiently and effectively. Our spatial memory essentially assembles object
instances back into a pseudo "image" representation that is easy to be fed into
another ConvNet for object-object context reasoning. This leads to a new
sequential reasoning architecture where image and memory are processed in
parallel to obtain detections which update the memory again. We show our SMN
direction is promising as it provides 2.2\% improvement over baseline Faster
RCNN on the COCO dataset so far.

本文提出了一种新的时空记忆网络 (Spatial Memory Network) 框架，其中空间记忆将目标实例聚合回代表简单 “图像” 的伪图像表示，易于输入到另一个 ConvNet 中进行目标间上下文推理，使得目标检测性能提高了 2.2％。