The advancement of transformer neural networks has significantly elevated the
capabilities of sentence similarity models, particularly in creating effective
vector representations of natural language inputs. However, these models face
notable challenges in domain-specific contexts, especially in highly
specialized scientific sub-fields. Traditional methods often struggle in this
regime, either overgeneralizing similarities within a niche or being overly
sensitive to minor differences, resulting in inaccurate text classification and
subpar vector representation. In an era where retrieval augmentation and search
are increasingly crucial, precise and concise numerical representations are
essential. In this paper, we target this issue by assembling niche datasets
using co-citations as a similarity metric, focusing on biomedical domains. We
employ two key strategies for fine-tuning state-of-the-art models: 1.
Domain-specific Fine-Tuning, which tailors pretrained models to a single
domain, and 2. Universal Applicability with Mixture of Experts (MoE), adapting
pretrained models with enforced routing for multiple domains simultaneously.
Our training approach emphasizes the use of abstracts for faster training,
incorporating Multiple Negative Rankings loss for efficient contrastive
learning. Notably, our MoE variants, equipped with $N$ experts, achieve the
efficacy of $N$ individual models, heralding a new era of versatile,
One-Size-Fits-All transformer networks for various tasks. This methodology
marks significant advancements in scientific text classification metrics and
holds promise for enhancing vector database search and compilation.

我们针对高度专业化的科学子领域中，传统方法不能很好地进行文本分类和矢量表示的问题，提出了使用共引作为相似度度量的专业数据集，并结合领域特定微调和专家混合的通用适应性方法，实现了在多个任务上具有高效性的通用 Transformer 网络，显著推进了科学文本分类指标的发展，并有望增强矢量数据库的搜索和编译。