In the era of billion-parameter-sized Language Models (LMs), start-ups have
to follow trends and adapt their technology accordingly. Nonetheless, there are
open challenges since the development and deployment of large models comes with
a need for high computational resources and has economical consequences. In
this work, we follow the steps of the R&D group of a modern legal-tech start-up
and present important insights on model development and deployment. We start
from ground zero by pre-training multiple domain-specific multi-lingual LMs
which are a better fit to contractual and regulatory text compared to the
available alternatives (XLM-R). We present benchmark results of such models in
a half-public half-private legal benchmark comprising 5 downstream tasks
showing the impact of larger model size. Lastly, we examine the impact of a
full-scale pipeline for model compression which includes: a) Parameter Pruning,
b) Knowledge Distillation, and c) Quantization: The resulting models are much
more efficient without sacrificing performance at large.

本研究探讨了如何从零开始构建多个特定领域的多语言语言模型，并测试了它们在法律语料库上的效果。同时，研究者还讨论了模型的压缩方法，以减少计算资源和经济成本并保持模型性能。