Scaling up neural models has yielded significant advancements in a wide array
of tasks, particularly in language generation. Previous studies have found that
the performance of neural models frequently adheres to predictable scaling
laws, correlated with factors such as training set size and model size. This
insight is invaluable, especially as large-scale experiments grow increasingly
resource-intensive. Yet, such scaling law has not been fully explored in dense
retrieval due to the discrete nature of retrieval metrics and complex
relationships between training data and model sizes in retrieval tasks. In this
study, we investigate whether the performance of dense retrieval models follows
the scaling law as other neural models. We propose to use contrastive
log-likelihood as the evaluation metric and conduct extensive experiments with
dense retrieval models implemented with different numbers of parameters and
trained with different amounts of annotated data. Results indicate that, under
our settings, the performance of dense retrieval models follows a precise
power-law scaling related to the model size and the number of annotations.
Additionally, we examine scaling with prevalent data augmentation methods to
assess the impact of annotation quality, and apply the scaling law to find the
best resource allocation strategy under a budget constraint. We believe that
these insights will significantly contribute to understanding the scaling
effect of dense retrieval models and offer meaningful guidance for future
research endeavors.

文章研究了稠密检索模型的性能是否遵循与其他神经模型相同的缩放定律，并提出使用对比对数似然作为评估指标，并通过大量实验证明，在我们设置下，稠密检索模型的性能遵循与模型大小和注释数量相关的明确的幂律缩放。此外，文章还通过流行的数据增强方法来检验缩放效果，以评估注释质量的影响，并应用缩放定律来找到在预算限制下的最佳资源分配策略，这些研究结果对于理解稠密检索模型的缩放效应并为未来的研究提供有意义的指导具有重要意义。