Two-time-scale optimization is a framework introduced in Zeng et al. (2024)
that abstracts a range of policy evaluation and policy optimization problems in
reinforcement learning (RL). Akin to bi-level optimization under a particular
type of stochastic oracle, the two-time-scale optimization framework has an
upper level objective whose gradient evaluation depends on the solution of a
lower level problem, which is to find the root of a strongly monotone operator.
In this work, we propose a new method for solving two-time-scale optimization
that achieves significantly faster convergence than the prior arts. The key
idea of our approach is to leverage an averaging step to improve the estimates
of the operators in both lower and upper levels before using them to update the
decision variables. These additional averaging steps eliminate the direct
coupling between the main variables, enabling the accelerated performance of
our algorithm. We characterize the finite-time convergence rates of the
proposed algorithm under various conditions of the underlying objective
function, including strong convexity, convexity, Polyak-Lojasiewicz condition,
and general non-convexity. These rates significantly improve over the
best-known complexity of the standard two-time-scale stochastic approximation
algorithm. When applied to RL, we show how the proposed algorithm specializes
to novel online sample-based methods that surpass or match the performance of
the existing state of the art. Finally, we support our theoretical results with
numerical simulations in RL.

提出了一种用于解决两时间尺度优化问题的新方法，通过利用平均化步骤改善算子的估计，消除了主要变量之间的直接耦合，从而大大加快了收敛速度，并在强凸性、凸性、Polyak-Lojasiewicz 条件和一般非凸性等各种情况下改进了传统两时间尺度随机逼近算法的复杂性，该算法在强化学习中表现出色，超越或与现有的最先进方法相匹配，并通过强化学习中的数值模拟验证了理论结果。

快速二次时间尺度随机梯度法及其在强化学习中的应用

Fast Two-Time-Scale Stochastic Gradient Method with Applications in  Reinforcement Learning

Many machine learning problems encode their data as a matrix with a possibly
very large number of rows and columns. In several applications like
neuroscience, image compression or deep reinforcement learning, the principal
subspace of such a matrix provides a useful, low-dimensional representation of
individual data. Here, we are interested in determining the $d$-dimensional
principal subspace of a given matrix from sample entries, i.e. from small
random submatrices. Although a number of sample-based methods exist for this
problem (e.g. Oja's rule \citep{oja1982simplified}), these assume access to
full columns of the matrix or particular matrix structure such as symmetry and
cannot be combined as-is with neural networks \citep{baldi1989neural}. In this
paper, we derive an algorithm that learns a principal subspace from sample
entries, can be applied when the approximate subspace is represented by a
neural network, and hence can be scaled to datasets with an effectively
infinite number of rows and columns. Our method consists in defining a loss
function whose minimizer is the desired principal subspace, and constructing a
gradient estimate of this loss whose bias can be controlled. We complement our
theoretical analysis with a series of experiments on synthetic matrices, the
MNIST dataset \citep{lecun2010mnist} and the reinforcement learning domain
PuddleWorld \citep{sutton1995generalization} demonstrating the usefulness of
our approach.

本文基于样本输入数据推导出一种算法，可以学习如何从具有潜在主子空间的矩阵中进行分类或降维，适用于神经网络，可以有效地扩展到无限数量的行和列上。

一种学习主子空间的新型随机梯度下降算法

A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspaces

Domain adaptation has become a prominent problem setting in machine learning
and related fields. This review asks the question: how can a classifier learn
from a source domain and generalize to a target domain? We present a
categorization of approaches, divided into, what we refer to as, sample-based,
feature-based and inference-based methods. Sample-based methods focus on
weighting individual observations during training based on their importance to
the target domain. Feature-based methods revolve around on mapping, projecting
and representing features such that a source classifier performs well on the
target domain and inference-based methods incorporate adaptation into the
parameter estimation procedure, for instance through constraints on the
optimization procedure. Additionally, we review a number of conditions that
allow for formulating bounds on the cross-domain generalization error. Our
categorization highlights recurring ideas and raises questions important to
further research.

本篇综述旨在回答分类器如何从源域学习并推广到目标域，分类方法包括基于样本的方法、基于特征的方法和基于推理的方法，文章还探讨了进一步研究所需的一些问题和条件。