One of the main challenges of multimodal learning is the need to combine
heterogeneous modalities (e.g., video, audio, text). For example, video and
audio are obtained at much higher rates than text and are roughly aligned in
time. They are often not synchronized with text, which comes as a global
context, e.g., a title, or a description. Furthermore, video and audio inputs
are of much larger volumes, and grow as the video length increases, which
naturally requires more compute dedicated to these modalities and makes
modeling of long-range dependencies harder.
We here decouple the multimodal modeling, dividing it into separate, focused
autoregressive models, processing the inputs according to the characteristics
of the modalities. We propose a multimodal model, called Mirasol3B, consisting
of an autoregressive component for the time-synchronized modalities (audio and
video), and an autoregressive component for the context modalities which are
not necessarily aligned in time but are still sequential. To address the
long-sequences of the video-audio inputs, we propose to further partition the
video and audio sequences in consecutive snippets and autoregressively process
their representations. To that end, we propose a Combiner mechanism, which
models the audio-video information jointly within a timeframe. The Combiner
learns to extract audio and video features from raw spatio-temporal signals,
and then learns to fuse these features producing compact but expressive
representations per snippet.
Our approach achieves the state-of-the-art on well established multimodal
benchmarks, outperforming much larger models. It effectively addresses the high
computational demand of media inputs by both learning compact representations,
controlling the sequence length of the audio-video feature representations, and
modeling their dependencies in time.

我们提出了一种多模态模型 Mirasol3B，通过自回归机制，将多媒体输入分别处理并建模，同时使用分段和合并机制来处理视频和音频序列的长依赖性，从而实现了在多模态基准测试上的最佳结果。

Mirasol3B：面向时间对齐和上下文多模态自回归模型

Mirasol3B: A Multimodal Autoregressive model for time-aligned and  contextual modalities

COMpression with Bayesian Implicit NEural Representations (COMBINER) is a
recent data compression method that addresses a key inefficiency of previous
Implicit Neural Representation (INR)-based approaches: it avoids quantization
and enables direct optimization of the rate-distortion performance. However,
COMBINER still has significant limitations: 1) it uses factorized priors and
posterior approximations that lack flexibility; 2) it cannot effectively adapt
to local deviations from global patterns in the data; and 3) its performance
can be susceptible to modeling choices and the variational parameters'
initializations. Our proposed method, Robust and Enhanced COMBINER
(RECOMBINER), addresses these issues by 1) enriching the variational
approximation while maintaining its computational cost via a linear
reparameterization of the INR weights, 2) augmenting our INRs with learnable
positional encodings that enable them to adapt to local details and 3)
splitting high-resolution data into patches to increase robustness and
utilizing expressive hierarchical priors to capture dependency across patches.
We conduct extensive experiments across several data modalities, showcasing
that RECOMBINER achieves competitive results with the best INR-based methods
and even outperforms autoencoder-based codecs on low-resolution images at low
bitrates.

COMBINER 是一种最近的数据压缩方法，它通过避免量化和直接优化速率 - 失真性能来解决以前基于隐式神经表示 (INR) 的方法的一个关键低效性。然而，COMBINER 仍然存在显著的限制：1）它使用了缺乏灵活性的分解先验和后验近似；2）它不能有效地适应数据中的局部偏差和全局模式；3）它的性能容易受到模型选择和变分参数初始化的影响。我们提出的方法，Robust and Enhanced COMBINER (RECOMBINER)，通过以下方式解决了这些问题：1）通过线性重参数化 INR 权重，丰富变分近似并保持其计算成本；2）通过学习可调整的位置编码来增强 INR，使其能够适应局部细节；3）将高分辨率数据分割成补丁以增加鲁棒性，并利用具有表达力的分层先验来捕捉补丁间的依赖关系。我们在多个数据模态上进行了大量实验证明，RECOMBINER 取得了与最佳 INR 方法相媲美的结果，并且在低比特率的低分辨率图像上甚至胜过自动编码器的编解码器。

RECOMBINER: 基于贝叶斯隐式神经表征的稳健增强压缩

RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit  Neural Representations

Transformers provide a class of expressive architectures that are extremely
effective for sequence modeling. However, the key limitation of transformers is
their quadratic memory and time complexity $\mathcal{O}(L^2)$ with respect to
the sequence length in attention layers, which restricts application in
extremely long sequences. Most existing approaches leverage sparsity or
low-rank assumptions in the attention matrix to reduce cost, but sacrifice
expressiveness. Instead, we propose Combiner, which provides full attention
capability in each attention head while maintaining low computation and memory
complexity. The key idea is to treat the self-attention mechanism as a
conditional expectation over embeddings at each location, and approximate the
conditional distribution with a structured factorization. Each location can
attend to all other locations, either via direct attention, or through indirect
attention to abstractions, which are again conditional expectations of
embeddings from corresponding local regions. We show that most sparse attention
patterns used in existing sparse transformers are able to inspire the design of
such factorization for full attention, resulting in the same sub-quadratic cost
($\mathcal{O}(L\log(L))$ or $\mathcal{O}(L\sqrt{L})$). Combiner is a drop-in
replacement for attention layers in existing transformers and can be easily
implemented in common frameworks. An experimental evaluation on both
autoregressive and bidirectional sequence tasks demonstrates the effectiveness
of this approach, yielding state-of-the-art results on several image and text
modeling tasks.

提出了一种名为 Combiner 的替代 transformers 中的 attention layers 的方法，其通过利用结构化分解的条件分布来近似自我关注机制，实现了每个关注头的完全关注能力，同时保持低计算和内存复杂性。在自回归和双向序列任务上的实验评估表明，该方法有效性高，可以产生多个图像和文本建模任务的最新技术结果。