Understanding the limitations and weaknesses of state-of-the-art models in artificial intelligence is crucial for their improvement and responsible application. In this research, we focus on CLIP, a model renowned for its integration of vision and language processing. Our objective is to uncover recurring problems and blind spots in CLIP's image comprehension. By delving into both the commonalities and disparities between CLIP and human image understanding, we augment our comprehension of these models' capabilities. Through our analysis, we reveal significant discrepancies in CLIP's interpretation of images compared to human perception, shedding light on areas requiring improvement. Our methodologies, the Discrepancy Analysis Framework (DAF) and the Transformative Caption Analysis for CLIP (TCAC), enable a comprehensive evaluation of CLIP's performance. We identify 14 systemic faults, including Action vs. Stillness confusion, Failure to identify the direction of movement or positioning of objects in the image, Hallucination of Water-like Features, Misattribution of Geographic Context, among others. By addressing these limitations, we lay the groundwork for the development of more accurate and nuanced image embedding models, contributing to advancements in artificial intelligence.

我们的研究旨在揭示CLIP模型在图像理解方面存在的问题和盲点，通过对比CLIP与人类图像理解的共性与差异，我们发现CLIP在图像解释上存在与人类感知相比的显著差异，包括行动与静止的混淆、无法识别图像中的运动方向或物体位置、产生类似水的特征的幻觉，以及对地理环境的错误归因等14个系统性问题。通过解决这些限制，我们为开发更准确细致的图像嵌入模型奠定了基础，推动了人工智能的发展。

揭秘故障现象：对CLIP图像编码错误进行深入研究