We study the problem of smooth imitation learning for online sequence
prediction, where the goal is to train a policy that can smoothly imitate
demonstrated behavior in a dynamic and continuous environment in response to
online, sequential context input. Since the mapping from context to behavior is
often complex, we take a learning reduction approach to reduce smooth imitation
learning to a regression problem using complex function classes that are
regularized to ensure smoothness. We present a learning meta-algorithm that
achieves fast and stable convergence to a good policy. Our approach enjoys
several attractive properties, including being fully deterministic, employing
an adaptive learning rate that can provably yield larger policy improvements
compared to previous approaches, and the ability to ensure stable convergence.
Our empirical results demonstrate significant performance gains over previous
approaches.

研究在线序列预测的平滑模仿学习问题，通过学习缩减的方法将其降低到回归问题，并采用正则化复杂函数类以确保平滑度。提出了一种元算法，实现快速稳定地收敛到好的策略，相比于以前的方法，具有全部确定性、自适应学习率等优点，并能保证稳定收敛，实证结果证明了性能得到了显著的提高。

在线序列预测的平滑模仿学习

Smooth Imitation Learning for Online Sequence Prediction

Online sequence prediction is the problem of predicting the next element of a
sequence given previous elements. This problem has been extensively studied in
the context of individual sequence prediction, where no prior assumptions are
made on the origin of the sequence. Individual sequence prediction algorithms
work quite well for long sequences, where the algorithm has enough time to
learn the temporal structure of the sequence. However, they might give poor
predictions for short sequences. A possible remedy is to rely on the general
model of prediction with expert advice, where the learner has access to a set
of $r$ experts, each of which makes its own predictions on the sequence. It is
well known that it is possible to predict almost as well as the best expert if
the sequence length is order of $\log(r)$. But, without firm prior knowledge on
the problem, it is not clear how to choose a small set of {\em good} experts.
In this paper we describe and analyze a new algorithm that learns a good set of
experts using a training set of previously observed sequences. We demonstrate
the merits of our approach by applying it on the task of click prediction on
the web.

本文介绍一种在线学习算法，利用先前观察到的数据训练一组有效的专家，以在短序列上进行点击预测。