Object detection is a computer vision task of predicting a set of bounding
boxes and category labels for each object of interest in a given image. The
category is related to a linguistic symbol such as 'dog' or 'person' and there
should be relationships among them. However the object detector only learns to
classify the categories and does not treat them as the linguistic symbols.
Multi-modal models often use the pre-trained object detector to extract object
features from the image, but the models are separated from the detector and the
extracted visual features does not change with their linguistic input. We
rethink the object detection as a vision-and-language reasoning task. We then
propose targeted detection task, where detection targets are given by a natural
language and the goal of the task is to detect only all the target objects in a
given image. There are no detection if the target is not given. Commonly used
modern object detectors have many hand-designed components like anchor and it
is difficult to fuse the textual inputs into the complex pipeline. We thus
propose Language-Targeted Detector (LTD) for the targeted detection based on a
recently proposed Transformer-based detector. LTD is a encoder-decoder
architecture and our conditional decoder allows the model to reason about the
encoded image with the textual input as the linguistic context. We evaluate
detection performances of LTD on COCO object detection dataset and also show
that our model improves the detection results with the textual input grounding
to the visual object.

本文探讨了一种将物体检测转化为视觉与语言推理任务的方法，并提出了一种基于 Transformer 的编码器 - 解码器体系结构下的语言目标检测器（LTD），该方法将文本输入作为语言上下文进行推理，扩展了现有物体检测器的分类功能。通过对 COCO 数据集的检测表现进行评估，证明了 LTD 不仅可以改善物体检测结果，还可以通过文本输入与视觉对象的基础连接，更好地推理目标检测任务。

仅依据指定的语言目标进行物体检测

Detect Only What You Specify : Object Detection with Linguistic Target

Recent years have witnessed a trend of applying context frames to boost the
performance of object detection as video object detection. Existing methods
usually aggregate features at one stroke to enhance the feature. These methods,
however, usually lack spatial information from neighboring frames and suffer
from insufficient feature aggregation. To address the issues, we perform a
progressive way to introduce both temporal information and spatial information
for an integrated enhancement. The temporal information is introduced by the
temporal feature aggregation model (TFAM), by conducting an attention mechanism
between the context frames and the target frame (i.e., the frame to be
detected). Meanwhile, we employ a Spatial Transition Awareness Model (STAM) to
convey the location transition information between each context frame and
target frame. Built upon a transformer-based detector DETR, our PTSEFormer also
follows an end-to-end fashion to avoid heavy post-processing procedures while
achieving 88.1% mAP on the ImageNet VID dataset. Codes are available at
this https URL.

本论文提出了一种基于 PTSEFormer 的对象检测方法，该方法引入了上下文框架、时间信息、空间信息来提高模型性能，其通过注意力机制与视觉特征的集成实现时间信息的引入，同时通过 Spatial Transition Awareness Model 实现了上下文特征之间的空间信息的集成，最终在 ImageNet VID 数据集上取得了 88.1% mAP 的表现。