This paper focuses on activity retrieval from a video query in an imbalanced
scenario. In current query-by-activity-video literature, a common assumption is
that all activities have sufficient labelled examples when learning an
embedding. This assumption does however practically not hold, as only a portion
of activities have many examples, while other activities are only described by
few examples. In this paper, we propose a visual-semantic embedding network
that explicitly deals with the imbalanced scenario for activity retrieval. Our
network contains two novel modules. The visual alignment module performs a
global alignment between the input video and fixed-sized visual bank
representations for all activities. The semantic module performs an alignment
between the input video and fixed-sized semantic activity representations. By
matching videos with both visual and semantic activity representations that are
of equal size over all activities, we no longer ignore infrequent activities
during retrieval. Experiments on a new imbalanced activity retrieval benchmark
show the effectiveness of our approach for all types of activities.

本文针对非平衡场景中的视频查询中的活动检索进行研究，提出了一种视觉 - 语义嵌入网络，该网络包含两个新模块：视觉对齐模块通过全局对齐输入视频和所有活动的固定大小视觉库表示之间的关系，语义模块在输入视频和固定大小的语义活动表示之间进行对齐。通过匹配具有相等规模的视觉和语义活动表示的视频，我们不再在检索过程中忽略不常见的活动。实验结果表明，我们的方法在各类活动的非平衡活动检索基准上具有有效性。

野外活动视频查询

Query by Activity Video in the Wild

Large Language Models (LLMs) have consistently showcased remarkable
generalization capabilities when applied to various language tasks.
Nonetheless, harnessing the full potential of LLMs for Radiology Report
Generation (R2Gen) still presents a challenge, stemming from the inherent
disparity in modality between LLMs and the R2Gen task. To bridge this gap
effectively, we propose R2GenGPT, which is a novel solution that aligns visual
features with the word embedding space of LLMs using an efficient visual
alignment module. This innovative approach empowers the previously static LLM
to seamlessly integrate and process image information, marking a step forward
in optimizing R2Gen performance. R2GenGPT offers the following benefits. First,
it attains state-of-the-art (SOTA) performance by training only the lightweight
visual alignment module while freezing all the parameters of LLM. Second, it
exhibits high training efficiency, as it requires the training of an
exceptionally minimal number of parameters while achieving rapid convergence.
By employing delta tuning, our model only trains 5M parameters (which
constitute just 0.07\% of the total parameter count) to achieve performance
close to the SOTA levels. Our code is available at
this https URL

为了有效地弥合 LLMs 和 R2Gen 任务之间的模态差异，提出了一种新颖的解决方案 R2GenGPT，通过高效的视觉对齐模块将视觉特征与 LLMs 的词嵌入空间对齐，使得以前静态的 LLMs 能够无缝集成和处理图像信息，从而优化 R2Gen 性能。