Recently, multimodal large language models have made significant advancements
in video understanding tasks. However, their ability to understand unprocessed
long videos is very limited, primarily due to the difficulty in supporting the
enormous memory overhead. Although existing methods achieve a balance between
memory and information by aggregating frames, they inevitably introduce the
severe hallucination issue. To address this issue, this paper constructs a
comprehensive hallucination mitigation pipeline based on existing MLLMs.
Specifically, we use the CLIP Score to guide the frame sampling process with
questions, selecting key frames relevant to the question. Then, We inject
question information into the queries of the image Q-former to obtain more
important visual features. Finally, during the answer generation stage, we
utilize chain-of-thought and in-context learning techniques to explicitly
control the generation of answers. It is worth mentioning that for the
breakpoint mode, we found that image understanding models achieved better
results than video understanding models. Therefore, we aggregated the answers
from both types of models using a comparison mechanism. Ultimately, We achieved
84.2\% and 62.9\% for the global and breakpoint modes respectively on the
MovieChat dataset, surpassing the official baseline model by 29.1\% and 24.1\%.
Moreover the proposed method won the third place in the CVPR LOVEU 2024
Long-Term Video Question Answering Challenge. The code is avaiable at
this https URL

本文提出了一种基于现有多模态大型语言模型的全面幻觉缓解方法，通过使用 CLIP 分数指导带有问题的帧采样过程，并将问题信息注入到图像 Q-former 的查询中以获取更重要的视觉特征，最终在 MovieChat 数据集上取得了 84.2％和 62.9％的全局和断点模式准确率，超过了官方基线模型的 29.1％和 24.1％，在 CVPR LOVEU 2024 长期视频问答挑战中获得了第三名。

幻觉减轻促进长期视频理解

Hallucination Mitigation Prompts Long-term Video Understanding

Artificial Intelligence Generative Content (AIGC) technologies have
significantly influenced the remote sensing domain, particularly in the realm
of image generation. However, remote sensing image editing, an equally vital
research area, has not garnered sufficient attention. Different from
text-guided editing in natural images, which relies on extensive text-image
paired data for semantic correlation, the application scenarios of remote
sensing image editing are often extreme, such as forest on fire, so it is
difficult to obtain sufficient paired samples. At the same time, the lack of
remote sensing semantics and the ambiguity of text also restrict the further
application of image editing in remote sensing field. To solve above problems,
this letter proposes a diffusion based method to fulfill stable and
controllable remote sensing image editing with text guidance. Our method avoids
the use of a large number of paired image, and can achieve good image editing
results using only a single image. The quantitative evaluation system including
CLIP score and subjective evaluation metrics shows that our method has better
editing effect on remote sensing images than the existing image editing model.

本文提出了一种基于扩散方法的稳定可控的远程感知图像编辑方法，避免了大量配对图像的使用，并且通过量化评估系统和主观评估指标表明，该方法在远程感知图像编辑方面的效果优于现有的图像编辑模型。

探究基于文本引导的遥感图像单幅编辑

Exploring Text-Guided Single Image Editing for Remote Sensing Images

The quality of pre-training data plays a critical role in the performance of
foundation models. Popular foundation models often design their own recipe for
data filtering, which makes it hard to analyze and compare different data
filtering approaches. DataComp is a new benchmark dedicated to evaluating
different methods for data filtering. This paper describes our learning and
solution when participating in the DataComp challenge. Our filtering strategy
includes three stages: single-modality filtering, cross-modality filtering, and
data distribution alignment. We integrate existing methods and propose new
solutions, such as computing CLIP score on horizontally flipped images to
mitigate the interference of scene text, using vision and language models to
retrieve training samples for target downstream tasks, rebalancing the data
distribution to improve the efficiency of allocating the computational budget,
etc. We slice and dice our design choices, provide in-depth analysis, and
discuss open questions. Our approach outperforms the best method from the
DataComp paper by over 4% on the average performance of 38 tasks and by over 2%
on ImageNet.

本论文介绍了在参加 DataComp 挑战时的学习和解决方案，通过三个阶段的筛选策略与已有方法整合以及提出新的解决方案，如使用水平翻转图像计算 CLIP 分数以减轻场景文字的干扰、利用视觉和语言模型检索目标下游任务的训练样本，重新平衡数据分布以提高计算预算分配效率等，获得了比 DataComp 论文中最佳方法平均性能提高 4% 的成绩，以及在 ImageNet 上提高 2% 以上的成绩。