Observations with distributed sensors are essential in analyzing a series of
human and machine activities (referred to as 'events' in this paper) in complex
and extensive real-world environments. This is because the information obtained
from a single sensor is often missing or fragmented in such an environment;
observations from multiple locations and modalities should be integrated to
analyze events comprehensively. However, a learning method has yet to be
established to extract joint representations that effectively combine such
distributed observations. Therefore, we propose Guided Masked sELf-Distillation
modeling (Guided-MELD) for inter-sensor relationship modeling. The basic idea
of Guided-MELD is to learn to supplement the information from the masked sensor
with information from other sensors needed to detect the event. Guided-MELD is
expected to enable the system to effectively distill the fragmented or
redundant target event information obtained by the sensors without being overly
dependent on any specific sensors. To validate the effectiveness of the
proposed method in novel tasks of distributed multimedia sensor event analysis,
we recorded two new datasets that fit the problem setting: MM-Store and
MM-Office. These datasets consist of human activities in a convenience store
and an office, recorded using distributed cameras and microphones. Experimental
results on these datasets show that the proposed Guided-MELD improves event
tagging and detection performance and outperforms conventional inter-sensor
relationship modeling methods. Furthermore, the proposed method performed
robustly even when sensors were reduced.

提出 Guided-MELD 模型来进行传感器间的关系建模，通过学习补充过程中通过遮蔽传感器无法获取的信息，有效地提取分散的目标事件信息，从而改善事件标记和检测性能。

分布式多媒体传感器事件分析的引导式屏蔽自蒸馏建模

Guided Masked Self-Distillation Modeling for Distributed Multimedia  Sensor Event Analysis

Building joint representations across images and text is an essential step
for tasks such as Visual Question Answering and Video Question Answering. In
this work, we find that the representations must not only jointly capture
features from both modalities but should also be diverse for better
generalization performance. To this end, we propose joint vision-language
representation learning by diversifying the tokenization learning process,
enabling tokens that are sufficiently disentangled from each other to be
learned from both modalities. We observe that our approach outperforms the
baseline models in a majority of settings and is competitive with
state-of-the-art methods.

建立跨图片和文本的联合表示是视觉问答和视频问答等任务的重要步骤。本研究发现，这些表示不仅必须同时捕捉两种模态的特征，而且还应具有多样性，以获得更好的泛化性能。为此，提出了通过多样化分词学习过程的联合视觉语言表示学习，可以学习来自两种模态的足够解开的词元。观察到我们的方法在大多数情况下优于基线模型，并且与最先进的方法竞争力强。

多元化的联合视觉 - 语言标记化学习

Diversifying Joint Vision-Language Tokenization Learning

Multimodal sentiment analysis is a core research area that studies speaker
sentiment expressed from the language, visual, and acoustic modalities. The
central challenge in multimodal learning involves inferring joint
representations that can process and relate information from these modalities.
However, existing work learns joint representations by requiring all modalities
as input and as a result, the learned representations may be sensitive to noisy
or missing modalities at test time. With the recent success of sequence to
sequence (Seq2Seq) models in machine translation, there is an opportunity to
explore new ways of learning joint representations that may not require all
input modalities at test time. In this paper, we propose a method to learn
robust joint representations by translating between modalities. Our method is
based on the key insight that translation from a source to a target modality
provides a method of learning joint representations using only the source
modality as input. We augment modality translations with a cycle consistency
loss to ensure that our joint representations retain maximal information from
all modalities. Once our translation model is trained with paired multimodal
data, we only need data from the source modality at test time for final
sentiment prediction. This ensures that our model remains robust from
perturbations or missing information in the other modalities. We train our
model with a coupled translation-prediction objective and it achieves new
state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI,
ICT-MMMO, and YouTube. Additional experiments show that our model learns
increasingly discriminative joint representations with more input modalities
while maintaining robustness to missing or perturbed modalities.

该论文提出了一种基于模态翻译的方法来学习稳健的联合表示，并使用循环一致性损失来保留所有模态的最大信息，从而实现了多模态情感分析方面的最新成果。