High relevance of retrieved and re-ranked items to the search query is the
cornerstone of successful product search, yet measuring relevance of items to
queries is one of the most challenging tasks in product information retrieval,
and quality of product search is highly influenced by the precision and scale
of available relevance-labelled data. In this paper, we present an array of
techniques for leveraging Large Language Models (LLMs) for automating the
relevance judgment of query-item pairs (QIPs) at scale. Using a unique dataset
of multi-million QIPs, annotated by human evaluators, we test and optimize
hyper parameters for finetuning billion-parameter LLMs with and without Low
Rank Adaption (LoRA), as well as various modes of item attribute concatenation
and prompting in LLM finetuning, and consider trade offs in item attribute
inclusion for quality of relevance predictions. We demonstrate considerable
improvement over baselines of prior generations of LLMs, as well as
off-the-shelf models, towards relevance annotations on par with the human
relevance evaluators. Our findings have immediate implications for the growing
field of relevance judgment automation in product search.

基于 Large Language Models (LLMs) 的技术，通过自动化查询 - 商品对的相关性判断，改善产品搜索的相关性预测精度，对商品搜索的相关判断自动化领域具有重要影响。