Embedding models are integral to AI applications like semantic search,
personalized recommendations, and retrieval augmented generation for LLMs,
necessitating high-quality training data. However, the limited scalability of
manual data curation prompts the need for automated methods to ensure data
integrity. Traditional unsupervised triplet mining automates training data
generation, crucial for embedding model training, yet inadvertently injects
biases and noise, thereby degrading model performance. Addressing this, we
introduce GISTEmbed, a novel strategy that enhances in-batch negative selection
during contrastive training through a guide model. This approach departs from
reliance on random sampling and equal utility assumption of batch negatives,
significantly reducing noise from data quality issues and improving model
fine-tuning. Benchmarked against the Massive Text Embedding Benchmark (MTEB),
GISTEmbed showcases consistent performance improvements across various model
sizes and achieves state-of-the-art results in select categories. This
framework enables significant enhancements for smaller models by leveraging the
capabilities of powerful yet resource-intensive large models. GISTEmbed can
potentially revolutionize the creation of highly efficient, smaller models,
democratizing access to advanced AI technologies. Making these technologies
more accessible and cost-effective, especially for applications constrained by
resources, significantly expands the impact and accessibility of
state-of-the-art AI solutions across diverse sectors.

通过引入 GISTEmbed，在对比训练中通过导向模型增强批内负例选择，从而显著减少数据质量问题引入的噪声，并改善模型微调，从而提供了对规模较小模型的显著改进的框架，借助资源密集型的大型模型的能力，GISTEmbed 有可能革新高效且规模较小模型的创建方法，使得先进的 AI 技术更加可用和具有可承担性，从而在不同领域大大扩展了最先进的 AI 解决方案的影响和可及性。