The future success of the Navy will depend, in part, on artificial
intelligence. In practice, many artificially intelligent algorithms, and in
particular deep learning models, rely on continual learning to maintain
performance in dynamic environments. The software requires adaptation to
maintain its initial level of performance in unseen situations. However, if not
monitored properly, continual learning may lead to several issues including
catastrophic forgetting in which a trained model forgets previously learned
tasks when being retrained on new data. The authors created a new framework for
safely performing continual learning with the goal of pairing this safety
framework with a deep learning computer vision algorithm to allow for safe and
high-performing automatic deck tracking on carriers and amphibious assault
ships. The safety framework includes several features, such as an ensemble of
convolutional neural networks to perform image classification, a manager to
record confidences and determine the best answer from the ensemble, a model of
the environment to predict when the system may fail to meet minimum performance
metrics, a performance monitor to log system and domain performance and check
against requirements, and a retraining component to update the ensemble and
manager to maintain performance. The authors validated the proposed method
using extensive simulation studies based on dynamic image classification. The
authors showed the safety framework could probabilistically detect out of
distribution data. The results also show the framework can detect when the
system is no longer performing safely and can significantly extend the working
envelope of an image classifier.

作者创建了一个新的安全框架，并验证了该框架在动态图像分类中的有效性，该框架可以用于安全地进行自动甲板跟踪，并显著扩展图像分类器的工作范围。

部署的持续学习系统保证

Assurance for Deployed Continual Learning Systems

The proven efficacy of learning-based control schemes strongly motivates
their application to robotic systems operating in the physical world. However,
guaranteeing correct operation during the learning process is currently an
unresolved issue, which is of vital importance in safety-critical systems. We
propose a general safety framework based on Hamilton-Jacobi reachability
methods that can work in conjunction with an arbitrary learning algorithm. The
method exploits approximate knowledge of the system dynamics to guarantee
constraint satisfaction while minimally interfering with the learning process.
We further introduce a Bayesian mechanism that refines the safety analysis as
the system acquires new evidence, reducing initial conservativeness when
appropriate while strengthening guarantees through real-time validation. The
result is a least-restrictive, safety-preserving control law that intervenes
only when (a) the computed safety guarantees require it, or (b) confidence in
the computed guarantees decays in light of new observations. We prove
theoretical safety guarantees combining probabilistic and worst-case analysis
and demonstrate the proposed framework experimentally on a quadrotor vehicle.
Even though safety analysis is based on a simple point-mass model, the
quadrotor successfully arrives at a suitable controller by policy-gradient
reinforcement learning without ever crashing, and safely retracts away from a
strong external disturbance introduced during flight.

这篇研究论文就如何在机器人系统的学习过程中保证安全性提出了一种新的框架，综合运用了学习控制方法、Hamilton-Jacobi 可达性方法和贝叶斯机制，通过概率和最坏情况分析提供了理论安全保障，并在四旋翼飞行器上实验室验证了其有效性。