We conducted extensive experiments on domain adaptation of the
Meta-Llama-3-70B-Instruct model on SEC data, exploring its performance on both
general and domain-specific benchmarks. Our focus included continual
pre-training (CPT) and model merging, aiming to enhance the model's
domain-specific capabilities while mitigating catastrophic forgetting. Through
this study, we evaluated the impact of integrating financial regulatory data
into a robust language model and examined the effectiveness of our model
merging techniques in preserving and improving the model's instructive
abilities. The model is accessible at hugging face:
this https URL, arcee-ai/Llama-3-SEC-Base.
This is an intermediate checkpoint of our final model, which has seen 20B
tokens so far. The full model is still in the process of training. This is a
preprint technical report with thorough evaluations to understand the entire
process.

在金融监管数据集上，我们对 Meta-Llama-3-70B-Instruct 模型的领域适应性进行了广泛的实验，探索其在通用和特定领域基准上的性能。我们关注了持续预训练（CPT）和模型合并，旨在增强模型的特定领域能力，同时减轻灾难性遗忘。通过本研究，我们评估了将金融监管数据集整合到一个健壮的语言模型中的影响，并检验了我们的模型合并技术在保护和提高模型的指导能力方面的有效性。