This paper proposes a novel contrastive learning framework, called FOCAL, for
extracting comprehensive features from multimodal time-series sensing signals
through self-supervised training. Existing multimodal contrastive frameworks
mostly rely on the shared information between sensory modalities, but do not
explicitly consider the exclusive modality information that could be critical
to understanding the underlying sensing physics. Besides, contrastive
frameworks for time series have not handled the temporal information locality
appropriately. FOCAL solves these challenges by making the following
contributions: First, given multimodal time series, it encodes each modality
into a factorized latent space consisting of shared features and private
features that are orthogonal to each other. The shared space emphasizes feature
patterns consistent across sensory modalities through a modal-matching
objective. In contrast, the private space extracts modality-exclusive
information through a transformation-invariant objective. Second, we propose a
temporal structural constraint for modality features, such that the average
distance between temporally neighboring samples is no larger than that of
temporally distant samples. Extensive evaluations are performed on four
multimodal sensing datasets with two backbone encoders and two classifiers to
demonstrate the superiority of FOCAL. It consistently outperforms the
state-of-the-art baselines in downstream tasks with a clear margin, under
different ratios of available labels. The code and self-collected dataset are
available at this https URL

该论文提出了一种名为 FOCAL 的新型对比学习框架，通过自监督训练从多模态时间序列感知信号中提取全面的特征。FOCAL 通过引入共享特征和私有特征的分解潜空间，以在感官模式间保留一致的特征模式并提取独占的信息以及引入时间结构约束来处理挑战。在四个多模态感知数据集上进行了广泛的评估，证明了 FOCAL 在下游任务中始终优于现有基准，并在不同标签可用比例下取得了明显的优势。