Interactions with virtual assistants typically start with a predefined
trigger phrase followed by the user command. To make interactions with the
assistant more intuitive, we explore whether it is feasible to drop the
requirement that users must begin each command with a trigger phrase. We
explore this task in three ways: First, we train classifiers using only
acoustic information obtained from the audio waveform. Second, we take the
decoder outputs of an automatic speech recognition (ASR) system, such as 1-best
hypotheses, as input features to a large language model (LLM). Finally, we
explore a multimodal system that combines acoustic and lexical features, as
well as ASR decoder signals in an LLM. Using multimodal information yields
relative equal-error-rate improvements over text-only and audio-only models of
up to 39% and 61%. Increasing the size of the LLM and training with low-rank
adaption leads to further relative EER reductions of up to 18% on our dataset.

虚拟助手的交互通常以预定义的触发短语作为开端，我们探索是否可以放弃用户必须以触发短语开始每个指令的要求。通过三种方式进行实验：首先，只使用从音频波形中获得的声学信息来训练分类器；其次，将自动语音识别（ASR）系统的解码器输出，如 1 最佳假设，作为大型语言模型（LLM）的输入特征；最后，探索结合声学和词汇特征以及 ASR 解码器信号的多模态系统。使用多模态信息相对于仅文本和仅音频的模型，在等误差率上可以获得高达 39% 和 61% 的改进。增加 LLM 的规模，并使用低秩适应来进行训练，在我们的数据集上进一步降低了相对误差率高达 18%。

多模态方法在大型语言模型中的设备导向语音检测

A Multimodal Approach to Device-Directed Speech Detection with Large  Language Models

Interactions with virtual assistants typically start with a trigger phrase
followed by a command. In this work, we explore the possibility of making these
interactions more natural by eliminating the need for a trigger phrase. Our
goal is to determine whether a user addressed the virtual assistant based on
signals obtained from the streaming audio recorded by the device microphone. We
address this task by combining 1-best hypotheses and decoder signals from an
automatic speech recognition system with acoustic representations from an audio
encoder as input features to a large language model (LLM). In particular, we
are interested in data and resource efficient systems that require only a small
amount of training data and can operate in scenarios with only a single frozen
LLM available on a device. For this reason, our model is trained on 80k or less
examples of multimodal data using a combination of low-rank adaptation and
prefix tuning. We compare the proposed system to unimodal baselines and show
that the multimodal approach achieves lower equal-error-rates (EERs), while
using only a fraction of the training data. We also show that low-dimensional
specialized audio representations lead to lower EERs than high-dimensional
general audio representations.

通过消除触发短语的需要，本研究探索了使用流式音频录制的设备麦克风记录的信号来确定用户是否在与虚拟助手进行交流，通过将语音识别系统的最佳假设和解码器信号与音频编码器的声学表示结合为大型语言模型的输入特征来实现这一目标。使用低秩适应和前缀调整的组合来对 80,000 个或更少的多模式数据示例进行训练，我们将所提出的系统与单模式基线进行比较，结果表明多模式方法实现了更低的等错误率（EER），同时仅使用了训练数据的一小部分，并且低维度的专门音频表示比高维度的普通音频表示导致更低的错误率（EER）。

多模态数据和资源高效的设备导向语音检测与大型基础模型

Multimodal Data and Resource Efficient Device-Directed Speech Detection  with Large Foundation Models

Adversarial attacks alter NLP model predictions by perturbing test-time
inputs. However, it is much less understood whether, and how, predictions can
be manipulated with small, concealed changes to the training data. In this
work, we develop a new data poisoning attack that allows an adversary to
control model predictions whenever a desired trigger phrase is present in the
input. For instance, we insert 50 poison examples into a sentiment model's
training set that causes the model to frequently predict Positive whenever the
input contains "James Bond". Crucially, we craft these poison examples using a
gradient-based procedure so that they do not mention the trigger phrase. We
also apply our poison attack to language modeling ("Apple iPhone" triggers
negative generations) and machine translation ("iced coffee" mistranslated as
"hot coffee"). We conclude by proposing three defenses that can mitigate our
attack at some cost in prediction accuracy or extra human annotation.

本研究开发了一种新的数据污染攻击方法，能够在训练数据中插入少量样本并控制模型预测结果，其中包含一个特定的强制词，同时提出了三种缓解该攻击的防御策略。