Traditional defect classification approaches are facing with two barriers.
(1) Insufficient training data and unstable data quality. Collecting sufficient
defective sample is expensive and time-costing, consequently leading to dataset
variance. It introduces the difficulty on recognition and learning. (2)
Over-dependence on visual modality. When the image pattern and texture is
monotonic for all defect classes in a given dataset, the performance of
conventional AOI system cannot be guaranteed. In scenarios where image quality
is compromised due to mechanical failures or when defect information is
inherently difficult to discern, the performance of deep models cannot be
guaranteed. A main question is, "how to solve those two problems when they
occur at the same time?" The feasible strategy is to explore another feature
within dataset and combine an eminent vision-language model (VLM) and
Large-Language model (LLM) with their astonishing zero-shot capability. In this
work, we propose the special ASE dataset, including rich data description
recorded on image, for defect classification, but the defect feature is uneasy
to learn directly. Secondly, We present the prompting for VLM-LLM against
defect classification with the proposed ASE dataset to activate extra-modality
feature from images to enhance performance. Then, We design the novel
progressive feature alignment (PFA) block to refine image-text feature to
alleviate the difficulty of alignment under few-shot scenario. Finally, the
proposed Cross-modality attention fusion (CMAF) module can effectively fuse
different modality feature. Experiment results have demonstrated our method's
effectiveness over several defect classification methods for the ASE dataset.

我们提出了 ASE 数据集，其中包含丰富的图像数据描述，用于缺陷分类，但缺陷特征难以直接学习。我们通过使用专门的视觉语言模型（VLM）和大型语言模型（LLM）的惊人零样本能力，为差异分类问题提供了解决方案。我们通过在图像中激活额外的多模态特征以提高性能，并设计了逐步特征对齐（PFA）块来改善少样本情况下的对齐困难。最后，我们提出的跨模态注意力融合（CMAF）模块可以有效地融合不同的模态特征。实验证明了我们的方法在 ASE 数据集上相对于其他缺陷分类方法的有效性。