While contrastive language image pretraining (CLIP) have exhibited impressive
performance by learning highly semantic and generalized representations, recent
works have exposed a fundamental drawback in its syntactic properties, that
includes interpreting fine-grained attributes, actions, spatial relations,
states, and details that require compositional reasoning. One reason for this
is that natural captions often do not capture all the visual details of a
scene. This leads to unaddressed visual concepts being misattributed to the
wrong words. And the pooled image and text features, ends up acting as a bag of
words, hence losing the syntactic information. In this work, we ask: Is it
possible to enhance CLIP's fine-grained and syntactic abilities without
compromising its semantic properties? We show that this is possible by adapting
CLIP efficiently on a high-quality, comprehensive, and relatively small
dataset. We demonstrate our adaptation strategy on VidSitu, a video situation
recognition dataset annotated with verbs and rich semantic role labels (SRL).
We use the SRL and verb information to create rule-based detailed captions,
making sure they capture most of the visual concepts. Combined with hard
negatives and hierarchical losses, these annotations allow us to learn a
powerful visual representation, dubbed Fine-Grained CLIP (FiGCLIP), that
preserves semantic understanding while being detail-oriented. We evaluate on
five diverse vision-language tasks in both fine-tuning and zero-shot settings,
achieving consistent improvements over the base CLIP model.

通过基于 VidSitu 数据集的细节导向字幕和层级损失，我们改进了 contrastive language image pretraining (CLIP) 模型，提高了其对细粒度和句法的理解能力，并在不同任务中取得了稳定的改进。

FiGCLIP: 细粒度 CLIP 适应通过密集标注视频

FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos

To understand how well a large language model captures certain semantic or
syntactic features, researchers typically apply probing classifiers. However,
the accuracy of these classifiers is critical for the correct interpretation of
the results. If a probing classifier exhibits low accuracy, this may be due
either to the fact that the language model does not capture the property under
investigation, or to shortcomings in the classifier itself, which is unable to
adequately capture the characteristics encoded in the internal representations
of the model. Consequently, for more effective diagnosis, it is necessary to
use the most accurate classifiers possible for a particular type of task.
Logistic regression on the output representation of the transformer neural
network layer is most often used to probing the syntactic properties of the
language model.
We show that using gradient boosting decision trees at the Knowledge Neuron
layer, i.e., at the hidden layer of the feed-forward network of the transformer
as a probing classifier for recognizing parts of a sentence is more
advantageous than using logistic regression on the output representations of
the transformer layer. This approach is also preferable to many other methods.
The gain in error rate, depending on the preset, ranges from 9-54%

使用梯度提升决策树作为语言模型中隐藏层的检测分类器，相对于逻辑回归于转换器层的输出表示，用于识别句子部分更具优势，并且比许多其他方法更可取，根据预置设置，错误率的增益范围为 9-54%。

知识树：基于知识神经元的梯度提升决策树作为探测分类器

Knowledge Trees: Gradient Boosting Decision Trees on Knowledge Neurons  as Probing Classifier

Transformers have supplanted recurrent models in a large number of NLP tasks.
However, the differences in their abilities to model different syntactic
properties remain largely unknown. Past works suggest that LSTMs generalize
very well on regular languages and have close connections with counter
languages. In this work, we systematically study the ability of Transformers to
model such languages as well as the role of its individual components in doing
so. We first provide a construction of Transformers for a subclass of counter
languages, including well-studied languages such as n-ary Boolean Expressions,
Dyck-1, and its generalizations. In experiments, we find that Transformers do
well on this subclass, and their learned mechanism strongly correlates with our
construction. Perhaps surprisingly, in contrast to LSTMs, Transformers do well
only on a subset of regular languages with degrading performance as we make
languages more complex according to a well-known measure of complexity. Our
analysis also provides insights on the role of self-attention mechanism in
modeling certain behaviors and the influence of positional encoding schemes on
the learning and generalization abilities of the model.

本文系统研究了 Transformers 模型在模拟正则语言和反计数语言中的能力，并探讨了其中各组件的作用，发现相比于 LSTMs，在某些行为建模上有一定优势，同时也揭示了自我注意机制和位置编码对模型学习和泛化能力的影响。