While Transformers underpin modern large language models (LMs), there is a
growing list of alternative architectures with new capabilities, promises, and
tradeoffs. This makes choosing the right LM architecture challenging.
Recently-proposed $\textit{hybrid architectures}$ seek a best-of-all-worlds
approach that reaps the benefits of all architectures. Hybrid design is
difficult for two reasons: it requires manual expert-driven search, and new
hybrids must be trained from scratch. We propose $\textbf{Manticore}$, a
framework that addresses these challenges. Manticore $\textit{automates the
design of hybrid architectures}$ while reusing pretrained models to create
$\textit{pretrained}$ hybrids. Our approach augments ideas from differentiable
Neural Architecture Search (NAS) by incorporating simple projectors that
translate features between pretrained blocks from different architectures. We
then fine-tune hybrids that combine pretrained models from different
architecture families -- such as the GPT series and Mamba -- end-to-end. With
Manticore, we enable LM selection without training multiple models, the
construction of pretrained hybrids from existing pretrained models, and the
ability to $\textit{program}$ pretrained hybrids to have certain capabilities.
Manticore hybrids outperform existing manually-designed hybrids, achieve strong
performance on Long Range Arena (LRA) tasks, and can improve on pretrained
transformers and state space models.

Manticore 是一个自动化设计混合体系结构的框架，通过重用预训练模型创建预训练混合体，实现了语言模型选择、预训练混合体的构建以及对预训练混合体进行编程的功能，并在长距离竞技场任务上取得强大的性能。

具备 MAD 技能的预训练混合模型

Pretrained Hybrids with MAD Skills

Exploring dense connectivity of convolutional operators establishes critical
"synapses" to communicate feature vectors from different levels and enriches
the set of transformations on Computer Vision applications. Yet, even with
heavy-machinery approaches such as Neural Architecture Search (NAS),
discovering effective connectivity patterns requires tremendous efforts due to
either constrained connectivity design space or a sub-optimal exploration
process induced by an unconstrained search space. In this paper, we propose
CSCO, a novel paradigm that fabricates effective connectivity of convolutional
operators with minimal utilization of existing design motifs and further
utilizes the discovered wiring to construct high-performing ConvNets. CSCO
guides the exploration via a neural predictor as a surrogate of the
ground-truth performance. We introduce Graph Isomorphism as data augmentation
to improve sample efficiency and propose a Metropolis-Hastings Evolutionary
Search (MH-ES) to evade locally optimal architectures and advance search
quality. Results on ImageNet show ~0.6% performance improvement over
hand-crafted and NAS-crafted dense connectivity. Our code is publicly
available.

通过最小化利用现有设计模式来创建卷积算子的有效连接，并利用发现的连接构建高性能的 ConvNets，本文提出了 CSCO，一个新颖的范例，通过神经预测器作为地面真实性能的代理引导搜索，引入图同构的数据增强来提高样本效率，并提出了 Metropolis-Hastings 进化搜索 (MH-ES) 来避免局部最优结构并提高搜索质量。在 ImageNet 上的结果显示，与手工和 NAS 制作的密集连接相比，性能提高了约 0.6%。

CSCO：卷积算子的连接性搜索

CSCO: Connectivity Search of Convolutional Operators

Cascading multiple pre-trained models is an effective way to compose an
end-to-end system. However, fine-tuning the full cascaded model is parameter
and memory inefficient and our observations reveal that only applying adapter
modules on cascaded model can not achieve considerable performance as
fine-tuning. We propose an automatic and effective adaptive learning method to
optimize end-to-end cascaded multi-task models based on Neural Architecture
Search (NAS) framework. The candidate adaptive operations on each specific
module consist of frozen, inserting an adapter and fine-tuning. We further add
a penalty item on the loss to limit the learned structure which takes the
amount of trainable parameters into account. The penalty item successfully
restrict the searched architecture and the proposed approach is able to search
similar tuning scheme with hand-craft, compressing the optimizing parameters to
8.7% corresponding to full fine-tuning on SLURP with an even better
performance.

利用神经架构搜索（NAS）框架提出一种自动且有效的自适应学习方法来优化端到端级联多任务模型，通过在每个特定模块上选择冻结、插入适配器和微调等候选自适应操作，并在损失函数中添加了一个惩罚项限制学习结构，成功地压缩优化参数至相当于全微调的 8.7% 并获得更好的性能。