For the task of image classification, neural networks primarily rely on visual patterns. In robust networks, we would expect for visually similar classes to be represented similarly. We consider the problem of when semantically similar classes are visually dissimilar, and when visual similarity is present among non-similar classes. We propose a data augmentation technique with the goal of better aligning semantically similar classes with arbitrary (non-visual) semantic relationships. We leverage recent work in diffusion-based semantic mixing to generate semantic hybrids of two classes, and these hybrids are added to the training set as augmented data. We evaluate whether the method increases semantic alignment by evaluating model performance on adversarially perturbed data, with the idea that it should be easier for an adversary to switch one class to a similarly represented class. Results demonstrate that there is an increase in alignment of semantically similar classes when using our proposed data augmentation method.

针对视觉相似度与语义相似度不匹配的问题，通过数据增强技术提出了一种混合语义增强方法，利用扩散式语义融合生成两类之间的语义混合，并将其作为扩展的数据集，实验证明该方法可以显著提高模型在语义相似类别的对齐性。

神经网络中语义和视觉对齐的差异处理