We devise an online learning algorithm -- titled Switching via Monotone
Adapted Regret Traces (SMART) -- that adapts to the data and achieves regret
that is instance optimal, i.e., simultaneously competitive on every input
sequence compared to the performance of the follow-the-leader (FTL) policy and
the worst case guarantee of any other input policy. We show that the regret of
the SMART policy on any input sequence is within a multiplicative factor
$e/(e-1) \approx 1.58$ of the smaller of: 1) the regret obtained by FTL on the
sequence, and 2) the upper bound on regret guaranteed by the given worst-case
policy. This implies a strictly stronger guarantee than typical
`best-of-both-worlds' bounds as the guarantee holds for every input sequence
regardless of how it is generated. SMART is simple to implement as it begins by
playing FTL and switches at most once during the time horizon to the worst-case
algorithm. Our approach and results follow from an operational reduction of
instance optimal online learning to competitive analysis for the ski-rental
problem. We complement our competitive ratio upper bounds with a fundamental
lower bound showing that over all input sequences, no algorithm can get better
than a $1.43$-fraction of the minimum regret achieved by FTL and the
minimax-optimal policy. We also present a modification of SMART that combines
FTL with a ``small-loss" algorithm to achieve instance optimality between the
regret of FTL and the small loss regret bound.

我们提出了一种在线学习算法 —— 通过单调适应性遗憾追踪（SMART）进行切换，它适应数据并实现了在每个输入序列上相对于领导者跟随（FTL）策略的表现和任何其他输入策略的最坏情况保证同时有效的遗憾，通过我们的算法，我们证明 SMART 政策在任何输入序列上的遗憾在与 FTL 获得的遗憾和给定最坏情况策略保证的遺憾上都在乘法因子 e/(e-1)≈1.58 的范围内，同时它是简单易实施的，并通过一种基本的分析方法证明了实例上在线学习相对于滑雪租赁问题的竞争分析的可行性，我们还提出了 SMART 的一个修改版本，通过将 FTL 与 “小损失” 算法相结合，实现了在 FTL 和小损失遗憾上的实例最优性。

实例最优在线学习的 SMART 方法

The SMART approach to instance-optimal online learning

Acquiring an accurate world model online for model-based reinforcement
learning (MBRL) is challenging due to data nonstationarity, which typically
causes catastrophic forgetting for neural networks (NNs). From the online
learning perspective, a Follow-The-Leader (FTL) world model is desirable, which
optimally fits all previous experiences at each round. Unfortunately, NN-based
models need re-training on all accumulated data at every interaction step to
achieve FTL, which is computationally expensive for lifelong agents. In this
paper, we revisit models that can achieve FTL with incremental updates.
Specifically, our world model is a linear regression model supported by
nonlinear random features. The linear part ensures efficient FTL update while
the nonlinear random feature empowers the fitting of complex environments. To
best trade off model capacity and computation efficiency, we introduce a
locality sensitive sparse encoding, which allows us to conduct efficient sparse
updates even with very high dimensional nonlinear features. We validate the
representation power of our encoding and verify that it allows efficient online
learning under data covariate shift. We also show, in the Dyna MBRL setting,
that our world models learned online using a single pass of trajectory data
either surpass or match the performance of deep world models trained with
replay and other continual learning methods.

在线学习是一项具有挑战性的任务，由于数据非稳态性，神经网络通常会导致灾难性遗忘，线性回归模型支持非线性随机特征，以实现高效的 Follow-The-Leader 更新，通过局部敏感稀疏编码实现高维非线性特征的高效稀疏更新，我们验证了编码的表达能力，并验证其在数据协变转移下实现了高效的在线学习，同时在 Dyna MBRL 环境中，我们的在线学习的世界模型与使用回放和其他持续学习方法训练的深度世界模型相匹配或超过了性能。

本地敏感稀疏编码用于在线学习世界模型

Locality Sensitive Sparse Encoding for Learning World Models Online

Follow-the-Leader (FTL) is an intuitive sequential prediction strategy that
guarantees constant regret in the stochastic setting, but has terrible
performance for worst-case data. Other hedging strategies have better
worst-case guarantees but may perform much worse than FTL if the data are not
maximally adversarial. We introduce the FlipFlop algorithm, which is the first
method that provably combines the best of both worlds.
As part of our construction, we develop AdaHedge, which is a new way of
dynamically tuning the learning rate in Hedge without using the doubling trick.
AdaHedge refines a method by Cesa-Bianchi, Mansour and Stoltz (2007), yielding
slightly improved worst-case guarantees. By interleaving AdaHedge and FTL, the
FlipFlop algorithm achieves regret within a constant factor of the FTL regret,
without sacrificing AdaHedge's worst-case guarantees.
AdaHedge and FlipFlop do not need to know the range of the losses in advance;
moreover, unlike earlier methods, both have the intuitive property that the
issued weights are invariant under rescaling and translation of the losses. The
losses are also allowed to be negative, in which case they may be interpreted
as gains.

介绍了一种名为 FlipFlop 算法的新方法，该方法将 Follow-the-Leader 算法和 AdaHedge 策略相结合，同时不会牺牲前者的强势表现和后者的最坏情况保证。