Recently, the success of pre-training in text domain has been fully extended
to vision, audio, and cross-modal scenarios. The proposed pre-training models
of different modalities are showing a rising trend of homogeneity in their
model structures, which brings the opportunity to implement different
pre-training models within a uniform framework. In this paper, we present
TencentPretrain, a toolkit supporting pre-training models of different
modalities. The core feature of TencentPretrain is the modular design. The
toolkit uniformly divides pre-training models into 5 components: embedding,
encoder, target embedding, decoder, and target. As almost all of common modules
are provided in each component, users can choose the desired modules from
different components to build a complete pre-training model. The modular design
enables users to efficiently reproduce existing pre-training models or build
brand-new one. We test the toolkit on text, vision, and audio benchmarks and
show that it can match the performance of the original implementations.

这篇论文介绍了 TencentPretrain，一个支持不同模态的预训练模型的工具包，其核心特性是模块化设计，用户可以选择所需组件构建完整的预训练模型，设计使用户能够有效地重现现有的预训练模型或构建全新模型，并测试表明其可以匹配原始实现的性能。