The flourishing ecosystem centered around voice personal assistants (VPA),
such as Amazon Alexa, has led to the booming of VPA apps. The largest app
market Amazon skills store, for example, hosts over 200,000 apps. Despite their
popularity, the open nature of app release and the easy accessibility of apps
also raise significant concerns regarding security, privacy and quality.
Consequently, various testing approaches have been proposed to systematically
examine VPA app behaviors. To tackle the inherent lack of a visible user
interface in the VPA app, two strategies are employed during testing, i.e.,
chatbot-style testing and model-based testing. The former often lacks effective
guidance for expanding its search space, while the latter falls short in
interpreting the semantics of conversations to construct precise and
comprehensive behavior models for apps. In this work, we introduce Elevate, a
model-enhanced large language model (LLM)-driven VUI testing framework. Elevate
leverages LLMs' strong capability in natural language processing to compensate
for semantic information loss during model-based VUI testing. It operates by
prompting LLMs to extract states from VPA apps' outputs and generate
context-related inputs. During the automatic interactions with the app, it
incrementally constructs the behavior model, which facilitates the LLM in
generating inputs that are highly likely to discover new states. Elevate
bridges the LLM and the behavior model with innovative techniques such as
encoding behavior model into prompts and selecting LLM-generated inputs based
on the context relevance. Elevate is benchmarked on 4,000 real-world Alexa
skills, against the state-of-the-art tester Vitas. It achieves 15% higher state
space coverage compared to Vitas on all types of apps, and exhibits significant
advancement in efficiency.

利用 Elevate，一种基于大型语言模型（LLM）驱动的语音用户界面（VUI）测试框架，可以通过借助 LLMs 在自然语言处理方面的强大能力来弥补基于模型的 VUI 测试中的语义信息损失，并通过创新技术将 LLM 与行为模型结合，以提高测试精确性和效率。

模型增强的基于 LLM 的 VUI 测试 VPA 应用程序

Model-Enhanced LLM-Driven VUI Testing of VPA Apps

With the advancement of DNNs into safety-critical applications, testing
approaches for such models have gained more attention. A current direction is
the search for and identification of systematic weaknesses that put safety
assumptions based on average performance values at risk. Such weaknesses can
take on the form of (semantically coherent) subsets or areas in the input space
where a DNN performs systematically worse than its expected average. However,
it is non-trivial to attribute the reason for such observed low performances to
the specific semantic features that describe the subset. For instance,
inhomogeneities within the data w.r.t. other (non-considered) attributes might
distort results. However, taking into account all (available) attributes and
their interaction is often computationally highly expensive. Inspired by
counterfactual explanations, we propose an effective and computationally cheap
algorithm to validate the semantic attribution of existing subsets, i.e., to
check whether the identified attribute is likely to have caused the degraded
performance. We demonstrate this approach on an example from the autonomous
driving domain using highly annotated simulated data, where we show for a
semantic segmentation model that (i) performance differences among the
different pedestrian assets exist, but (ii) only in some cases is the asset
type itself the reason for this reduction in the performance.

通过反事实解释的方法，提出一种有效且计算成本低的算法，以验证现有子集的语义归因，即检查已识别的属性是否可能导致性能下降。在自动驾驶领域的一个示例中展示了该方法，通过高度注释的模拟数据，表明在语义分割模型中存在不同的行人资源的性能差异，但只有在某些情况下，资源类型本身才是性能降低的原因。