Memory-based meta-learning is a technique for approximating Bayes-optimal
predictors. Under fairly general conditions, minimizing sequential prediction
error, measured by the log loss, leads to implicit meta-learning. The goal of
this work is to investigate how far this interpretation can be realized by
current sequence prediction models and training regimes. The focus is on
piecewise stationary sources with unobserved switching-points, which arguably
capture an important characteristic of natural language and action-observation
sequences in partially observable environments. We show that various types of
memory-based neural models, including Transformers, LSTMs, and RNNs can learn
to accurately approximate known Bayes-optimal algorithms and behave as if
performing Bayesian inference over the latent switching-points and the latent
parameters governing the data distribution within each segment.

本研究旨在探究当前的序列预测模型及训练方法对于基于内隐元学习的解释有多大实现能力，特别是对于具有未观测切换点的分段平稳信号的处理，发现使用记忆型神经模型，包括 Transformers, LSTMs 以及 RNNs 等，可以准确地近似知道的基于 Bayes 最优算法，表现出对于隐含的切换点和每个分段内的数据分布参数进行贝叶斯推断的能力。