In multimodal machine learning, multiple modalities of data (e.g., text and
images) are combined to facilitate the learning of a better machine learning
model, which remains applicable to a corresponding unimodal task (e.g., text
generation). Recently, multimodal machine learning has enjoyed huge empirical
success (e.g. GPT-4). Motivated to develop theoretical justification for this
empirical success, Lu (NeurIPS '23, ALT '24) introduces a theory of multimodal
learning, and considers possible separations between theoretical models of
multimodal and unimodal learning. In particular, Lu (ALT '24) shows a
computational separation, which is relevant to worst-case instances of the
learning task.
In this paper, we give a stronger average-case computational separation,
where for "typical" instances of the learning task, unimodal learning is
computationally hard, but multimodal learning is easy. We then question how
"organic" the average-case separation is. Would it be encountered in practice?
To this end, we prove that under natural conditions, any given computational
separation between average-case unimodal and multimodal learning tasks implies
a corresponding cryptographic key agreement protocol. We suggest to interpret
this as evidence that very strong computational advantages of multimodal
learning may arise infrequently in practice, since they exist only for the
"pathological" case of inherently cryptographic distributions. However, this
does not apply to possible (super-polynomial) statistical advantages.

多模式机器学习与单模式学习的理论模型、计算分离以及平均情况的计算分离是本篇论文的主要研究内容。