Human-guided robotic exploration is a useful approach to gathering
information at remote locations, especially those that might be too risky,
inhospitable, or inaccessible for humans. Maintaining common ground between the
remotely-located partners is a challenge, one that can be facilitated by
multi-modal communication. In this paper, we explore how participants utilized
multiple modalities to investigate a remote location with the help of a robotic
partner. Participants issued spoken natural language instructions and received
from the robot: text-based feedback, continuous 2D LIDAR mapping, and
upon-request static photographs. We noticed that different strategies were
adopted in terms of use of the modalities, and hypothesize that these
differences may be correlated with success at several exploration sub-tasks. We
found that requesting photos may have improved the identification and counting
of some key entities (doorways in particular) and that this strategy did not
hinder the amount of overall area exploration. Future work with larger samples
may reveal the effects of more nuanced photo and dialogue strategies, which can
inform the training of robotic agents. Additionally, we announce the release of
our unique multi-modal corpus of human-robot communication in an exploration
context: SCOUT, the Situated Corpus on Understanding Transactions.

通过研究人工和机器人合作探索远程地点时的多模态交流，我们发现不同的模态使用策略可能与多个探索子任务的成功相关，并且请求照片的策略可能改进了某些关键实体的识别和计数，未妨碍整体区域的探索。

多模式人机协作成功导航：分析和语料发布

Navigating to Success in Multi-Modal Human-Robot Collaboration: Analysis  and Corpus Release

Recent advances in computer vision and natural language processing have
naturally led to active research in multi-modal tasks, including Referring
Image Segmentation (RIS). Recent approaches have advanced the frontier of RIS
by impressive margins, but they require an additional pretraining stage on
external visual grounding datasets to achieve the state-of-the-art
performances. We attempt to break free from this requirement by effectively
adapting Contrastive Language-Image Pretraining (CLIP) to RIS. We propose a
novel framework that residually adapts frozen CLIP features to RIS with Fusion
Adapters and Backbone Adapters. Freezing CLIP preserves the backbone's rich,
general image-text alignment knowledge, whilst Fusion Adapters introduce
multi-modal communication and Backbone Adapters inject new knowledge useful in
solving RIS. Our method reaches a new state of the art on three major RIS
benchmarks. We attain such performance without additional pretraining and
thereby absolve the necessity of extra training and data preparation. Source
code and model weights will be available upon publication.

本文提出了一种基于 Contrastive Language-Image Pretraining (CLIP) 的新框架，通过 Fusion Adapters 和 Backbone Adapters 的方式对 RIS 进行有效的调整，实现了在三个重要的 RIS benchmarks 上的新的最高水平，且无需额外的预训练和数据准备。