Leveraging multimodal information from biosignals is vital for building a
comprehensive representation of people's physical and mental states. However,
multimodal biosignals often exhibit substantial distributional shifts between
pretraining and inference datasets, stemming from changes in task specification
or variations in modality compositions. To achieve effective pretraining in the
presence of potential distributional shifts, we propose a frequency-aware
masked autoencoder ($\texttt{bio}$FAME) that learns to parameterize the
representation of biosignals in the frequency space. $\texttt{bio}$FAME
incorporates a frequency-aware transformer, which leverages a fixed-size
Fourier-based operator for global token mixing, independent of the length and
sampling rate of inputs. To maintain the frequency components within each input
channel, we further employ a frequency-maintain pretraining strategy that
performs masked autoencoding in the latent space. The resulting architecture
effectively utilizes multimodal information during pretraining, and can be
seamlessly adapted to diverse tasks and modalities at test time, regardless of
input size and order. We evaluated our approach on a diverse set of transfer
experiments on unimodal time series, achieving an average of $\uparrow$5.5%
improvement in classification accuracy over the previous state-of-the-art.
Furthermore, we demonstrated that our architecture is robust in modality
mismatch scenarios, including unpredicted modality dropout or substitution,
proving its practical utility in real-world applications. Code will be
available soon.

利用多模式生物信号的多模式信息对人的身体和心理状态进行全面建模具有重要意义，但多模式生物信号往往在预训练和推断数据集之间存在明显的分布变化，我们提出了一种频率感知的掩码自编码器（bioFAME），它在频率空间中学习参数化生物信号的表示，并利用固定大小的基于傅里叶的操作符进行全局的分词混合，通过在潜在空间中进行掩码自编码以维持每个输入通道中的频率分量，实现了在预训练过程中对多模态信息的有效利用，并且可以在测试阶段无缝适应不同任务和模态，无论其输入长短和采样率如何，我们在一系列单模态时间序列的迁移实验中评估了我们的方法，在分类准确度上相较于之前的最先进方法平均提升了 5.5％，此外，我们还证明了我们的架构在模态不匹配的情况下具有稳健性，包括未预测的模态丢失或替换，证明了它在实际应用中的实用性。