Large-Language Models (LLMs) have shifted the paradigm of natural language
data processing. However, their black-boxed and probabilistic characteristics
can lead to potential risks in the quality of outputs in diverse LLM
applications. Recent studies have tested Quality Attributes (QAs), such as
robustness or fairness, of LLMs by generating adversarial input texts. However,
existing studies have limited their coverage of QAs and tasks in LLMs and are
difficult to extend. Additionally, these studies have only used one evaluation
metric, Attack Success Rate (ASR), to assess the effectiveness of their
approaches. We propose a MEtamorphic Testing for Analyzing LLMs (METAL)
framework to address these issues by applying Metamorphic Testing (MT)
techniques. This approach facilitates the systematic testing of LLM qualities
by defining Metamorphic Relations (MRs), which serve as modularized evaluation
metrics. The METAL framework can automatically generate hundreds of MRs from
templates that cover various QAs and tasks. In addition, we introduced novel
metrics that integrate the ASR method into the semantic qualities of text to
assess the effectiveness of MRs accurately. Through the experiments conducted
with three prominent LLMs, we have confirmed that the METAL framework
effectively evaluates essential QAs on primary LLM tasks and reveals the
quality risks in LLMs. Moreover, the newly proposed metrics can guide the
optimal MRs for testing each task and suggest the most effective method for
generating MRs.

大型语言模型（LLMs）已经改变了自然语言数据处理的范式。我们提出了一个名为 METAL 的框架，用于通过应用变异测试技术对 LLM 的质量进行系统测试，并引入了新的度量方法来准确评估 Metamorphic Relations（MRs）的有效性，从而有效评估了主要 LLM 任务的关键质量属性并揭示了 LLM 中的质量风险。

METAL: 大型语言模型质量分析的变形测试框架

METAL: Metamorphic Testing Framework for Analyzing Large-Language Model  Qualities

Knowledge engineering is the process of creating and maintaining
knowledge-producing systems. Throughout the history of computer science and AI,
knowledge engineering workflows have been widely used because high-quality
knowledge is assumed to be crucial for reliable intelligent agents. However,
the landscape of knowledge engineering has changed, presenting four challenges:
unaddressed stakeholder requirements, mismatched technologies, adoption
barriers for new organizations, and misalignment with software engineering
practices. In this paper, we propose to address these challenges by developing
a reference architecture using a mainstream software methodology. By studying
the requirements of different stakeholders and eras, we identify 23 essential
quality attributes for evaluating reference architectures. We assess three
candidate architectures from recent literature based on these attributes.
Finally, we discuss the next steps towards a comprehensive reference
architecture, including prioritizing quality attributes, integrating components
with complementary strengths, and supporting missing socio-technical
requirements. As this endeavor requires a collaborative effort, we invite all
knowledge engineering researchers and practitioners to join us.

本文提出通过使用主流软件方法论来开发参考架构来解决现代知识工程的四个挑战，包括未解决的利益相关方需求，不匹配的技术，新组织的采用障碍和与软件工程实践的不匹配。我们通过研究不同利益相关者和时代的要求，确定了 23 个评估参考架构所必需的基本质量属性，并根据这些属性评估了最近文献中的三种候选架构。最后，我们讨论了实现全面参考架构的下一步行动，包括优先考虑质量属性，整合具有互补优势的组件以及支持缺失的社会技术要求。由于这需要协作努力，因此我们邀请所有知识工程研究人员和实践者加入我们。