Recent advancements in pre-trained large foundation models (LFM) have yielded
significant breakthroughs across various domains, including natural language
processing and computer vision. These models have been particularly impactful
in the domain of medical diagnostic tasks. With abundant unlabeled data, an LFM
has been developed for fundus images using the Vision Transformer (VIT) and a
self-supervised learning framework. This LFM has shown promising performance in
fundus disease diagnosis across multiple datasets. On the other hand, deep
learning models have long been challenged by dataset quality issues, such as
image quality and dataset bias. To investigate the influence of data quality on
LFM, we conducted explorations in two fundus diagnosis tasks using datasets of
varying quality. Specifically, we explored the following questions: Is LFM more
robust to image quality? Is LFM affected by dataset bias? Can fine-tuning
techniques alleviate these effects? Our investigation found that LFM exhibits
greater resilience to dataset quality issues, including image quality and
dataset bias, compared to typical convolutional networks. Furthermore, we
discovered that overall fine-tuning is an effective adapter for LFM to mitigate
the impact of dataset quality issues.

使用预训练的大型基础模型（LFM）和自我监督学习框架为视网膜图像开发了一个 LFM，该模型在多个数据集上对视网膜疾病的诊断表现出良好的性能；此外，研究发现 LFM 对于数据集质量问题（包括图像质量和数据集偏差）表现出更强的韧性，而整体微调是减轻数据集质量问题影响的有效适应器。

大型基础模型在诊断中数据集质量依然令人关注吗？

Is Dataset Quality Still a Concern in Diagnosis Using Large Foundation  Model?

Pre-trained large foundation models play a central role in the recent surge
of artificial intelligence, resulting in fine-tuned models with remarkable
abilities when measured on benchmark datasets, standard exams, and
applications. Due to their inherent complexity, these models are not well
understood. While small adversarial inputs to such models are well known, the
structures of the representation space are not well characterized despite their
fundamental importance. In this paper, using the vision transformers as an
example due to the continuous nature of their input space, we show via analyses
and systematic experiments that the representation space consists of large
piecewise linear subspaces where there exist very different inputs sharing the
same representations, and at the same time, local normal spaces where there are
visually indistinguishable inputs having very different representations. The
empirical results are further verified using the local directional estimations
of the Lipschitz constants of the underlying models. Consequently, the
resulting representations change the results of downstream models, and such
models are subject to overgeneralization and with limited semantically
meaningful generalization capability.

基于视觉转换器作为例子，通过分析和系统实验，我们展示了表示空间由大型分段线性子空间和局部正常空间组成，存在着不同输入具有相同表示和在视觉上难以区分的不同表示的输入。这些经验结果进一步通过基础模型的局部方向估计来验证。因此，得到的表示会改变下游模型的结果，并且这样的模型容易过度泛化且具有有限的迁移能力。

视觉转换器嵌入空间的引人注目的等效结构

Intriguing Equivalence Structures of the Embedding Space of Vision  Transformers

Multimodal Named Entity Recognition (MNER) is a crucial task for information
extraction from social media platforms such as Twitter. Most current methods
rely on attention weights to extract information from both text and images but
are often unreliable and lack interpretability. To address this problem, we
propose incorporating uncertainty estimation into the MNER task, producing
trustworthy predictions. Our proposed algorithm models the distribution of each
modality as a Normal-inverse Gamma distribution, and fuses them into a unified
distribution with an evidential fusion mechanism, enabling hierarchical
characterization of uncertainties and promotion of prediction accuracy and
trustworthiness. Additionally, we explore the potential of pre-trained large
foundation models in MNER and propose an efficient fusion approach that
leverages their robust feature representations. Experiments on two datasets
demonstrate that our proposed method outperforms the baselines and achieves new
state-of-the-art performance.

提出一种结合不确定性评估的多模态命名实体识别框架，该算法模拟每种形式的分布并将它们融合成统一分布，以促进预测准确性和可信度。同时提出了一种利用大型预训练基础模型的高效融合方法，实验表明该方法优于基准并实现了新的最先进性能。