A long-standing goal of intelligent assistants such as AR glasses/robots has been to assist users in affordance-centric real-world scenarios, such as "how can I run the microwave for 1 minute?". However, there is still no clear task definition and suitable benchmarks. In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos and scripts to guide the user step-by-step. To support the task, we constructed AssistQ, a new dataset comprising 529 question-answer samples derived from 100 newly filmed first-person videos. Each question should be completed with multi-step guidances by inferring from visual details (e.g., buttons' position) and textural details (e.g., actions like press/turn). To address this unique task, we developed a Question-to-Actions (Q2A) model that significantly outperforms several baseline methods while still having large room for improvement. We expect our task and dataset to advance Egocentric AI Assistant's development. Our project page is available at: https://showlab.github.io/assistq

本文定义了一个新的任务，称为“面向 affordance 的问题驱动任务完成”，提出了一个名为 AssistQ 的数据集，并开发了一个名为 Q2A 的模型来解决该任务，并在 AssistQ 数据集上对其进行验证。结果表明，该模型明显优于几个 VQA 相关的基线，但仍有很大的提升空间。我们期望我们的任务和数据集能够推进自我中心 AI 助理的发展。

AssistQ: 以能力为中心、问题驱动的自我中心助手任务完成