Predicting where a person is looking is a complex task, requiring to
understand not only the person's gaze and scene content, but also the 3D scene
structure and the person's situation (are they manipulating? interacting or
observing others? attentive?) to detect obstructions in the line of sight or
apply attention priors that humans typically have when observing others. In
this paper, we hypothesize that identifying and leveraging such priors can be
better achieved through the exploitation of explicitly derived multimodal cues
such as depth and pose. We thus propose a modular multimodal architecture
allowing to combine these cues using an attention mechanism. The architecture
can naturally be exploited in privacy-sensitive situations such as surveillance
and health, where personally identifiable information cannot be released. We
perform extensive experiments on the GazeFollow and VideoAttentionTarget public
datasets, obtaining state-of-the-art performance and demonstrating very
competitive results in the privacy setting case.

本文提出了一种使用多模态线索，采用注意机制的模块化架构，以检测观察者线路中的障碍物并应用关注优先级，从而更好地实现识别和利用先验知识的方法，并在 GazeFollow 和 VideoAttentionTarget 公共数据集上展示了最先进的性能和隐私保护的竞争结果。

面向隐私敏感环境的视线目标预测模块化多模态架构

A Modular Multimodal Architecture for Gaze Target Prediction:  Application to Privacy-Sensitive Settings

Despite the importance of unsupervised object detection, to the best of our
knowledge, there is no previous work addressing this problem. One main issue,
widely known to the community, is that object boundaries derived only from 2D
image appearance are ambiguous and unreliable. To address this, we exploit
LiDAR clues to aid unsupervised object detection. By exploiting the 3D scene
structure, the issue of localization can be considerably mitigated. We further
identify another major issue, seldom noticed by the community, that the
long-tailed and open-ended (sub-)category distribution should be accommodated.
In this paper, we present the first practical method for unsupervised object
detection with the aid of LiDAR clues. In our approach, candidate object
segments based on 3D point clouds are firstly generated. Then, an iterative
segment labeling process is conducted to assign segment labels and to train a
segment labeling network, which is based on features from both 2D images and 3D
point clouds. The labeling process is carefully designed so as to mitigate the
issue of long-tailed and open-ended distribution. The final segment labels are
set as pseudo annotations for object detection network training. Extensive
experiments on the large-scale Waymo Open dataset suggest that the derived
unsupervised object detection method achieves reasonable accuracy compared with
that of strong supervision within the LiDAR visible range. Code shall be
released.

通过利用 LiDAR 数据辅助进行无监督的二维对象检测，同时采用基于 3D 点云和 2D 图像特征的迭代分割标签网络对候选对象进行标签生成和训练，从而解决了之前存在的问题，得到了较为合理的检测结果。