Videos carry rich visual information including object description, action,
interaction, etc., but the existing multimodal large language models (MLLMs)
fell short in referential understanding scenarios such as video-based
referring. In this paper, we present Artemis, an MLLM that pushes video-based
referential understanding to a finer level. Given a video, Artemis receives a
natural-language question with a bounding box in any video frame and describes
the referred target in the entire video. The key to achieving this goal lies in
extracting compact, target-specific video features, where we set a solid
baseline by tracking and selecting spatiotemporal features from the video. We
train Artemis on the newly established VideoRef45K dataset with 45K video-QA
pairs and design a computationally efficient, three-stage training procedure.
Results are promising both quantitatively and qualitatively. Additionally, we
show that \model can be integrated with video grounding and text summarization
tools to understand more complex scenarios. Code and data are available at
this https URL

基于视频的指代理解一直是多模态大型语言模型的一个挑战，本文提出了 Artemis，一个将视频指代理解提升到更精细层次的模型。通过提取紧凑、目标特定的视频特征，Artemis 能够根据视频中的自然语言问题和感兴趣目标的边界框完成整个视频的描述。通过设计高效的三阶段训练过程，并在新建的 VideoRef45K 数据集上进行训练，实验结果从定量和定性上都显示出良好的性能。此外，实验还展示了该模型能够与视频对准和文本摘要工具集成，以理解更复杂的场景。

阿尔忒弥斯：迈向复杂视频中的指代理解

Artemis: Towards Referential Understanding in Complex Videos

We propose a novel method for generating abstract art. First an autoencoder
is trained to encode and decode the style representations of images, which are
extracted from source images with a pretrained VGG network. Then, the decoder
component of the autoencoder is extracted and used as a generator in a GAN. The
generator works with an ensemble of discriminators. Each discriminator takes
different style representations of the same images, and the generator is
trained to create images that create convincing style representations in order
to deceive all of the generators. The generator is also trained to maximize a
diversity term. The resulting images had a surreal, geometric quality. We call
our approach ARTEMIS (ARTistic Encoder- Multi- Discriminators Including
Self-Attention), as it uses the self-attention layers and an encoder-decoder
architecture.

我们提出了一种新颖的生成抽象艺术的方法，通过训练自动编码器，提取源图像的风格表示，并使用生成对抗网络中的解码器作为生成器，结合多个鉴别器使生成器能够欺骗全部鉴别器，并最大化多样性项，最终生成具有超现实几何特性的图像。我们将其称为 ARTEMIS（艺术编码器 - 包括自注意力的多鉴别器），因为它使用了自注意力层和编码器 - 解码器架构。

ARTEMIS: 使用多个判别器的 GAN 生成艺术

ARTEMIS: Using GANs with Multiple Discriminators to Generate Art

We introduce a framework - Artemis - to tackle the problem of learning in a
distributed or federated setting with communication constraints and device
partial participation. Several workers (randomly sampled) perform the
optimization process using a central server to aggregate their computations. To
alleviate the communication cost, Artemis allows to compress the information
sent in both directions (from the workers to the server and conversely)
combined with a memory mechanism. It improves on existing algorithms that only
consider unidirectional compression (to the server), or use very strong
assumptions on the compression operator, and often do not take into account
devices partial participation. We provide fast rates of convergence (linear up
to a threshold) under weak assumptions on the stochastic gradients (noise's
variance bounded only at optimal point) in non-i.i.d. setting, highlight the
impact of memory for unidirectional and bidirectional compression, analyze
Polyak-Ruppert averaging. We use convergence in distribution to obtain a lower
bound of the asymptotic variance that highlights practical limits of
compression. We propose two approaches to tackle the challenging case of
devices partial participation and provide experimental results to demonstrate
the validity of our analysis.

本研究引入了一种名为阿尔忒弥斯的框架，旨在解决分布式或联邦学习中的通信限制和设备部分参与的问题。该框架可在双向压缩信息方面改善现有算法，并提供了快速收敛率和针对部分参与设备挑战的解决方案。