Building multimodal dialogue understanding capabilities situated in the
in-cabin context is crucial to enhance passenger comfort in autonomous vehicle
(AV) interaction systems. To this end, understanding passenger intents from
spoken interactions and vehicle vision systems is a crucial component for
developing contextual and visually grounded conversational agents for AV.
Towards this goal, we explore AMIE (Automated-vehicle Multimodal In-cabin
Experience), the in-cabin agent responsible for handling multimodal
passenger-vehicle interactions. In this work, we discuss the benefits of a
multimodal understanding of in-cabin utterances by incorporating
verbal/language input together with the non-verbal/acoustic and visual clues
from inside and outside the vehicle. Our experimental results outperformed
text-only baselines as we achieved improved performances for intent detection
with a multimodal approach.

本文探讨在自动驾驶交互系统中，构建多模态对话理解能力在车内情境下以提高乘客舒适度的重要性；通过将语言输入与车内外的非语言 / 声音和视觉线索结合起来，实现对车内话语的多模态理解，从而为 AV 开发上下文和视觉基础的对话代理提供支持；实验结果表明，多模态方法在意图检测方面胜过文本基准方法。

车内对话代理的乘客意图音视频理解

Audio-Visual Understanding of Passenger Intents for In-Cabin  Conversational Agents

Understanding passenger intents and extracting relevant slots are important
building blocks towards developing contextual dialogue systems for natural
interactions in autonomous vehicles (AV). In this work, we explored AMIE
(Automated-vehicle Multi-modal In-cabin Experience), the in-cabin agent
responsible for handling certain passenger-vehicle interactions. When the
passengers give instructions to AMIE, the agent should parse such commands
properly and trigger the appropriate functionality of the AV system. In our
current explorations, we focused on AMIE scenarios describing usages around
setting or changing the destination and route, updating driving behavior or
speed, finishing the trip and other use-cases to support various natural
commands. We collected a multi-modal in-cabin dataset with multi-turn dialogues
between the passengers and AMIE using a Wizard-of-Oz scheme via a realistic
scavenger hunt game activity. After exploring various recent Recurrent Neural
Networks (RNN) based techniques, we introduced our own hierarchical joint
models to recognize passenger intents along with relevant slots associated with
the action to be performed in AV scenarios. Our experimental results
outperformed certain competitive baselines and achieved overall F1 scores of
0.91 for utterance-level intent detection and 0.96 for slot filling tasks. In
addition, we conducted initial speech-to-text explorations by comparing
intent/slot models trained and tested on human transcriptions versus noisy
Automatic Speech Recognition (ASR) outputs. Finally, we compared the results
with single passenger rides versus the rides with multiple passengers.

本文讨论了自动驾驶汽车（AV）中的 AMIE，它是处理某些乘客 - 汽车交互的代理程序。我们集中探索了 AMIE 应用场景，建立了用于识别与实现乘客意图及相应插槽的分层联合模型，该模型已在多种自然命令支持方面取得较好表现。我们收集了多模态、多回合的乘客 - AMIE 对话数据集，并对喊出来的指令进行了文本可计算的构建。