The aim of this work is to establish how accurately a recent semantic-based
foveal active perception model is able to complete visual tasks that are
regularly performed by humans, namely, scene exploration and visual search.
This model exploits the ability of current object detectors to localize and
classify a large number of object classes and to update a semantic description
of a scene across multiple fixations. It has been used previously in scene
exploration tasks. In this paper, we revisit the model and extend its
application to visual search tasks. To illustrate the benefits of using
semantic information in scene exploration and visual search tasks, we compare
its performance against traditional saliency-based models. In the task of scene
exploration, the semantic-based method demonstrates superior performance
compared to the traditional saliency-based model in accurately representing the
semantic information present in the visual scene. In visual search experiments,
searching for instances of a target class in a visual field containing multiple
distractors shows superior performance compared to the saliency-driven model
and a random gaze selection algorithm. Our results demonstrate that semantic
information, from the top-down, influences visual exploration and search tasks
significantly, suggesting a potential area of research for integrating it with
traditional bottom-up cues.

通过使用语义信息，可以在场景探索和视觉搜索任务中有效地完成视觉任务，从而证明了语义模型在视觉任务中的优越性。

基于语义的人形机器视觉任务的主动感知技术与视点焦点传感器

Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal  Sensors

Image is usually taken for expressing some kinds of emotions or purposes,
such as love, celebrating Christmas. There is another better way that combines
the image and relevant song to amplify the expression, which has drawn much
attention in the social network recently. Hence, the automatic selection of
songs should be expected. In this paper, we propose to retrieve semantic
relevant songs just by an image query, which is named as the image2song
problem. Motivated by the requirements of establishing correlation in
semantic/content, we build a semantic-based song retrieval framework, which
learns the correlation between image content and lyric words. This model uses a
convolutional neural network to generate rich tags from image regions, a
recurrent neural network to model lyric, and then establishes correlation via a
multi-layer perceptron. To reduce the content gap between image and lyric, we
propose to make the lyric modeling focus on the main image content via a tag
attention. We collect a dataset from the social-sharing multimodal data to
study the proposed problem, which consists of (image, music clip, lyric)
triplets. We demonstrate that our proposed model shows noticeable results in
the image2song retrieval task and provides suitable songs. Besides, the
song2image task is also performed.

本文提出一种基于语义的图像检索语音的解决方案，通过深度学习模型生成标签，使用神经网络建模歌词，并基于多层感知器建立图像和歌词之间的语义相关性，实验结果显示出明显的检索表现力。