Distributed online learning has been proven extremely effective in solving
large-scale machine learning problems involving streaming data. However,
information sharing between learners in distributed learning also raises
concerns about the potential leakage of individual learners' sensitive data. To
mitigate this risk, differential privacy, which is widely regarded as the "gold
standard" for privacy protection, has been widely employed in many existing
results on distributed online learning. However, these results often face a
fundamental tradeoff between learning accuracy and privacy. In this paper, we
propose a locally differentially private gradient tracking based distributed
online learning algorithm that successfully circumvents this tradeoff. Our
analysis shows that the proposed algorithm converges in mean square to the
exact optimal solution while ensuring rigorous local differential privacy, with
the cumulative privacy budget guaranteed to be finite even when the number of
iterations tends to infinity. The algorithm is applicable even when the
communication graph among learners is directed. To the best of our knowledge,
this is the first result that simultaneously ensures learning accuracy and
rigorous local differential privacy in distributed online learning over
directed graphs. We evaluate our algorithm's performance by using multiple
benchmark machine-learning applications, including logistic regression on the
"Mushrooms" dataset and CNN-based image classification on the "MNIST" and
"CIFAR-10" datasets, respectively. The experimental results confirm that the
proposed algorithm outperforms existing counterparts in both training and
testing accuracies.

本研究提出了一种基于梯度追踪的本地差分隐私分布式在线学习算法，通过确保严格的本地差分隐私，该算法在均方意义下收敛于精确的最优解，即使在迭代次数趋向无穷的情况下，累积隐私预算也是有限的。在实验中，该算法在多个基准机器学习应用中表现良好，并且在训练和测试准确性方面均优于现有的对应方法。

定向图中分布式在线学习的本地差分隐私梯度跟踪

Locally Differentially Private Gradient Tracking for Distributed Online  Learning over Directed Graphs

This paper studies distributed online learning under Byzantine attacks. The
performance of an online learning algorithm is often characterized by
(adversarial) regret, which evaluates the quality of one-step-ahead
decision-making when an environment provides adversarial losses, and a
sublinear bound is preferred. But we prove that, even with a class of
state-of-the-art robust aggregation rules, in an adversarial environment and in
the presence of Byzantine participants, distributed online gradient descent can
only achieve a linear adversarial regret bound, which is tight. This is the
inevitable consequence of Byzantine attacks, even though we can control the
constant of the linear adversarial regret to a reasonable level. Interestingly,
when the environment is not fully adversarial so that the losses of the honest
participants are i.i.d. (independent and identically distributed), we show that
sublinear stochastic regret, in contrast to the aforementioned adversarial
regret, is possible. We develop a Byzantine-robust distributed online momentum
algorithm to attain such a sublinear stochastic regret bound. Extensive
numerical experiments corroborate our theoretical analysis.

本研究旨在研究分布式在线学习在拜占庭攻击下的表现。通过使用一类最先进的强健合并规则，我们证明，即使在存在 byzantine 参与者和对抗性环境下，分布式在线梯度下降也只能实现线性对抗性后悔界限，并且我们可以将线性对抗性后悔的常数控制在合理水平。有趣的是，当环境不是完全对抗性时，我们展示了亚线性随机后悔的可能性，采用了一种拜占庭式强健分布式在线动量算法来实现。

拜占庭容错的分布式在线学习：驯服恶意环境中的对手

Byzantine-Robust Distributed Online Learning: Taming Adversarial  Participants in An Adversarial Environment

Distributed online learning is gaining increased traction due to its unique
ability to process large-scale datasets and streaming data. To address the
growing public awareness and concern on privacy protection, plenty of private
distributed online learning algorithms have been proposed, mostly based on
differential privacy which has emerged as the ``gold standard" for privacy
protection. However, these algorithms often face the dilemma of trading
learning accuracy for privacy. By exploiting the unique characteristics of
online learning, this paper proposes an approach that tackles the dilemma and
ensures both differential privacy and learning accuracy in distributed online
learning. More specifically, while ensuring a diminishing expected
instantaneous regret, the approach can simultaneously ensure a finite
cumulative privacy budget, even on the infinite time horizon. To cater for the
fully distributed setting, we adopt the local differential-privacy framework
which avoids the reliance on a trusted data curator, and hence, provides
stronger protection than the classic ``centralized" (global) differential
privacy. To the best of our knowledge, this is the first algorithm that
successfully ensures both rigorous local differential privacy and learning
accuracy. The effectiveness of the proposed algorithm is evaluated using
machine learning tasks, including logistic regression on the ``Mushrooms" and
``Covtype" datasets and CNN based image classification on the ``MNIST" and
``CIFAR-10" datasets.

该论文提出了一种针对分布式在线学习的隐私保护算法，在保证微分隐私的同时提高算法的学习准确率，并采用本地微分隐私算法框架来避免对数据收集方的信任，有效地对包括逻辑回归和基于 CNN 的图像分类等任务进行了测试。