Deep neural networks have been used widely to learn the latent structure of
datasets, across modalities such as images, shapes, and audio signals. However,
existing models are generally modality-dependent, requiring custom
architectures and objectives to process different classes of signals. We
leverage neural fields to capture the underlying structure in image, shape,
audio and cross-modal audiovisual domains in a modality-independent manner. We
cast our task as one of learning a manifold, where we aim to infer a
low-dimensional, locally linear subspace in which our data resides. By
enforcing coverage of the manifold, local linearity, and local isometry, our
model -- dubbed GEM -- learns to capture the underlying structure of datasets
across modalities. We can then travel along linear regions of our manifold to
obtain perceptually consistent interpolations between samples, and can further
use GEM to recover points on our manifold and glean not only diverse
completions of input images, but cross-modal hallucinations of audio or image
signals. Finally, we show that by walking across the underlying manifold of
GEM, we may generate new samples in our signal domains. Code and additional
results are available at this https URL

通过利用神经网络学习数据集的潜在结构，构建 GEM 模型，实现跨模态音像的模态无关表达与样本生成