In this paper we introduce Shackleton as a generalized framework enabling the
application of linear genetic programming -- a technique under the umbrella of
evolutionary algorithms -- to a variety of use cases. We also explore here a
novel application for this class of methods: optimizing sequences of LLVM
optimization passes. The algorithm underpinning Shackleton is discussed, with
an emphasis on the effects of different features unique to the framework when
applied to LLVM pass sequences. Combined with analysis of different
hyperparameter settings, we report the results on automatically optimizing pass
sequences using Shackleton for two software applications at differing
complexity levels. Finally, we reflect on the advantages and limitations of our
current implementation and lay out a path for further improvements. These
improvements aim to surpass hand-crafted solutions with an automatic discovery
method for an optimal pass sequence.

本研究介绍了 Shackleton 框架，将线性遗传编程技术应用于 LLVM 优化传递的序列优化，实现自动化优化；结果表明，自动优化可以优于手工优化方案。

使用 Shackleton 优化 LLVM Pass 序列：一种线性遗传编程框架

Optimizing LLVM Pass Sequences with Shackleton: A Linear Genetic Programming Framework

The learning rate (LR) schedule is one of the most important hyper-parameters
needing careful tuning in training DNNs. However, it is also one of the least
automated parts of machine learning systems and usually costs significant
manual effort and computing. Though there are pre-defined LR schedules and
optimizers with adaptive LR, they introduce new hyperparameters that need to be
tuned separately for different tasks/datasets. In this paper, we consider the
question: Can we automatically tune the LR over the course of training without
human involvement? We propose an efficient method, AutoLRS, which automatically
optimizes the LR for each training stage by modeling training dynamics. AutoLRS
aims to find an LR applied to every $\tau$ steps that minimizes the resulted
validation loss. We solve this black-box optimization on the fly by Bayesian
optimization (BO). However, collecting training instances for BO requires a
system to evaluate each LR queried by BO's acquisition function for $\tau$
steps, which is prohibitively expensive in practice. Instead, we apply each
candidate LR for only $\tau'\ll\tau$ steps and train an exponential model to
predict the validation loss after $\tau$ steps. This mutual-training process
between BO and the loss-prediction model allows us to limit the training steps
invested in the BO search. We demonstrate the advantages and the generality of
AutoLRS through extensive experiments of training DNNs for tasks from diverse
domains using different optimizers. The LR schedules auto-generated by AutoLRS
lead to a speedup of $1.22\times$, $1.43\times$, and $1.5\times$ when training
ResNet-50, Transformer, and BERT, respectively, compared to the LR schedules in
their original papers, and an average speedup of $1.31\times$ over
state-of-the-art heavily-tuned LR schedules.

该研究提出了一种名为 AutoLRS 的自适应学习率优化方法，使用贝叶斯优化自动调整神经网络中的学习率，能够显著提高训练的效率和准确性。

AutoLRS: 基于贝叶斯优化的自动学习率调整

AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on  the Fly

Training and inference efficiency of deep neural networks highly rely on the
performance of tensor operators on hardware platforms. Manually optimizing
tensor operators has limitations in terms of supporting new operators or
hardware platforms. Therefore, automatically optimizing device code
configurations of tensor operators is getting increasingly attractive. However,
current methods for tensor operator optimization usually suffer from poor
sample-efficiency due to the combinatorial search space. In this work, we
propose a novel evolutionary method, OpEvo, which efficiently explores the
search spaces of tensor operators by introducing a topology-aware mutation
operation based on q-random walk to leverage the topological structures over
the search spaces. Our comprehensive experiment results show that compared with
state-of-the-art (SOTA) methods OpEvo can find the best configuration with the
lowest variance and least efforts in the number of trials and wall-clock time.
All code of this work is available online.

本文提出了一种基于拓扑结构的变异操作的进化方法 OpEvo，用于高效地探索张量运算符的搜索空间，其能够自动优化设备代码配置，相对于最先进的方法，能够找到具有最低方差和最少试验次数的最佳配置。