Deep Learning (DL) is rapidly maturing to the point that it can be used in
safety- and security-crucial applications. However, adversarial samples, which
are undetectable to the human eye, pose a serious threat that can cause the
model to misbehave and compromise the performance of such applications.
Addressing the robustness of DL models has become crucial to understanding and
defending against adversarial attacks. In this study, we perform comprehensive
experiments to examine the effect of adversarial attacks and defenses on
various model architectures across well-known datasets. Our research focuses on
black-box attacks such as SimBA, HopSkipJump, MGAAttack, and boundary attacks,
as well as preprocessor-based defensive mechanisms, including bits squeezing,
median smoothing, and JPEG filter. Experimenting with various models, our
results demonstrate that the level of noise needed for the attack increases as
the number of layers increases. Moreover, the attack success rate decreases as
the number of layers increases. This indicates that model complexity and
robustness have a significant relationship. Investigating the diversity and
robustness relationship, our experiments with diverse models show that having a
large number of parameters does not imply higher robustness. Our experiments
extend to show the effects of the training dataset on model robustness. Using
various datasets such as ImageNet-1000, CIFAR-100, and CIFAR-10 are used to
evaluate the black-box attacks. Considering the multiple dimensions of our
analysis, e.g., model complexity and training dataset, we examined the behavior
of black-box attacks when models apply defenses. Our results show that applying
defense strategies can significantly reduce attack effectiveness. This research
provides in-depth analysis and insight into the robustness of DL models against
various attacks, and defenses.

深度学习模型对抗攻击和防御的鲁棒性的综合实验研究表明，模型复杂度和鲁棒性之间存在显著关系，并且应用防御策略可以显著减少攻击效果。

从攻击到防御：对黑箱设置中的深度学习安全措施的洞察

From Attack to Defense: Insights into Deep Learning Security Measures in  Black-Box Settings

Deep neural networks, like many other machine learning models, have recently
been shown to lack robustness against adversarially crafted inputs. These
inputs are derived from regular inputs by minor yet carefully selected
perturbations that deceive machine learning models into desired
misclassifications. Existing work in this emerging field was largely specific
to the domain of image classification, since the high-entropy of images can be
conveniently manipulated without changing the images' overall visual
appearance. Yet, it remains unclear how such attacks translate to more
security-sensitive applications such as malware detection - which may pose
significant challenges in sample generation and arguably grave consequences for
failure.
In this paper, we show how to construct highly-effective adversarial sample
crafting attacks for neural networks used as malware classifiers. The
application domain of malware classification introduces additional constraints
in the adversarial sample crafting problem when compared to the computer vision
domain: (i) continuous, differentiable input domains are replaced by discrete,
often binary inputs; and (ii) the loose condition of leaving visual appearance
unchanged is replaced by requiring equivalent functional behavior. We
demonstrate the feasibility of these attacks on many different instances of
malware classifiers that we trained using the DREBIN Android malware data set.
We furthermore evaluate to which extent potential defensive mechanisms against
adversarial crafting can be leveraged to the setting of malware classification.
While feature reduction did not prove to have a positive impact, distillation
and re-training on adversarially crafted samples show promising results.

该研究针对深度神经网络在恶意软件分类中存在的漏洞，通过对对抗样本进行有效构建的攻击，探讨如何构建在样本生成方面更具挑战性的更安全的模型。该研究表明，在对抗样本的制作方面，恶意软件分类与计算机视觉领域之间存在巨大的差异。本文还评估了潜在的防御机制对恶意软件分类的影响，并发现对抗样本的蒸馏和重新训练可以带来很有前途的结果。