Testing is a major approach to ensuring the quality of deep learning (DL)
libraries. Existing testing techniques commonly adopt differential testing to
relieve the need for test oracle construction. However, these techniques are
limited in finding implementations that offer the same functionality and
generating diverse test inputs for differential testing. This paper introduces
DLLens, a novel differential testing technique for DL library testing. Our
insight is that APIs in different DL libraries are commonly designed to
accomplish various computations for the same set of published DL algorithms.
Although the mapping of these APIs is not often one-to-one, we observe that
their computations can be mutually simulated after proper composition and
adaptation. The use of these simulation counterparts facilitates differential
testing for the detection of functional DL library bugs. Leveraging the
insight, we propose DLLens as a novel mechanism that utilizes a large language
model (LLM) to synthesize valid counterparts of DL library APIs. To generate
diverse test inputs, DLLens incorporates a static analysis method aided by LLM
to extract path constraints from all execution paths in each API and its
counterpart's implementations. These path constraints are then used to guide
the generation of diverse test inputs. We evaluate DLLens on two popular DL
libraries, TensorFlow and PyTorch. Our evaluation shows that DLLens can
synthesize counterparts for more than twice as many APIs found by
state-of-the-art techniques on these libraries. Moreover, DLLens can extract
26.7% more constraints and detect 2.5 times as many bugs as state-of-the-art
techniques. DLLens has successfully found 56 bugs in recent TensorFlow and
PyTorch libraries. Among them, 41 are previously unknown, 39 of which have been
confirmed by developers after reporting, and 19 of those confirmed bugs have
been fixed by developers.

DLLens 是一种用于 DL 库测试的新型差分测试技术，使用大型语言模型 (LLM) 综合 DL 库 API 的有效对应，并使用静态分析方法辅助生成不同的测试输入，该技术可检测到更多的漏洞和修复。

DLLens：通过 LLM 协助综合测试深度学习库

DLLens: Testing Deep Learning Libraries via LLM-aided Synthesis

Recent deep learning (DL) applications are mostly built on top of DL
libraries. The quality assurance of these libraries is critical to the
dependable deployment of DL applications. Techniques have been proposed to
generate various DL models and apply them to test these libraries. However,
their test effectiveness is constrained by the diversity of layer API calls in
their generated DL models. Our study reveals that these techniques can cover at
most 34.1% layer inputs, 25.9% layer parameter values, and 15.6% layer
sequences. As a result, we find that many bugs arising from specific layer API
calls (i.e., specific layer inputs, parameter values, or layer sequences) can
be missed by existing techniques. Because of this limitation, we propose COMET
to effectively generate DL models with diverse layer API calls for DL library
testing. COMET: (1) designs a set of mutation operators and a coverage-based
search algorithm to diversify layer inputs, layer parameter values, and layer
sequences in DL models. (2) proposes a model synthesis method to boost the test
efficiency without compromising the layer API call diversity. Our evaluation
result shows that COMET outperforms baselines by covering twice as many layer
inputs (69.7% vs. 34.1%), layer parameter values (50.2% vs. 25.9%), and layer
sequences (39.0% vs. 15.6%) as those by the state-of-the-art. Moreover, COMET
covers 3.4% more library branches than those by existing techniques. Finally,
COMET detects 32 new bugs in the latest version of eight popular DL libraries,
including TensorFlow and MXNet, with 21 of them confirmed by DL library
developers and 7 of those confirmed bugs have been fixed by developers.

通过研究发现现有深度学习库测试技术的测试效果受到 API 调用多样性的限制，为了更加有效地生成深度学习模型以实现较高的覆盖率和发现更多的库错误，本文提出了一种基于突变操作符的模型生成算法 COMET，该算法可实现对 API 调用多样性的有效控制并发现了 32 个新的库错误。