In this paper, we introduce Robi Butler, a novel household robotic system that enables multimodal interactions with remote users. Building on the advanced communication interfaces, Robi Butler allows users to monitor the robot's status, send text or voice instructions, and select target objects by hand pointing. At the core of our system is a high-level behavior module, powered by Large Language Models (LLMs), that interprets multimodal instructions to generate action plans. These plans are composed of a set of open vocabulary primitives supported by Vision Language Models (VLMs) that handle both text and pointing queries. The integration of the above components allows Robi Butler to ground remote multimodal instructions in the real-world home environment in a zero-shot manner. We demonstrate the effectiveness and efficiency of this system using a variety of daily household tasks that involve remote users giving multimodal instructions. Additionally, we conducted a user study to analyze how multimodal interactions affect efficiency and user experience during remote human-robot interaction and discuss the potential improvements.

本研究提出了一种新型的家庭机器人系统Robi Butler，解决了传统家庭机器人的交互限制问题。该系统利用先进的通信接口，实现用户通过文本、语音和手指指向进行多模态指令的发送与执行，显示出优秀的高效性和用户体验。研究表明，Robi Butler能够有效地在无预训练环境下理解和执行远程指令，潜在地提升家庭机器人在日常任务中的应用价值。

Robi Butler：远程多模态家庭机器人助手