E-commerce sellers are recommended keyphrases based on their inventory on which they advertise to increase buyer engagement (clicks/sales). The relevance of advertiser keyphrases plays an important role in preventing the inundation of search systems with numerous irrelevant items that compete for attention in auctions, in addition to maintaining a healthy seller perception. In this work, we describe the shortcomings of training Advertiser keyphrase relevance filter models on click/sales/search relevance signals and the importance of aligning with human judgment, as sellers have the power to adopt or reject said keyphrase recommendations. In this study, we frame Advertiser keyphrase relevance as a complex interaction between 3 dynamical systems -- seller judgment, which influences seller adoption of our product, Advertising, which provides the keyphrases to bid on, and Search, who holds the auctions for the same keyphrases. This study discusses the practicalities of using human judgment via a case study at eBay Advertising and demonstrate that using LLM-as-a-judge en-masse as a scalable proxy for seller judgment to train our relevance models achieves a better harmony across the three systems -- provided that they are bound by a meticulous evaluation framework grounded in business metrics.

本研究解决了电商卖家在广告中推介关键短语时面临的相关性不足问题，强调了与人类判断对齐的重要性。通过将广告主关键短语相关性框架化为卖家判断、广告和搜索三者间的复杂动态交互，本研究提出了一种新的方法，利用大型语言模型作为卖家判断的可扩展代理，显著提升相关性模型的效果。此研究的关键发现是，在严谨的评估框架下，这种方法可以更好地协调卖家、广告和搜索三者的需求与期望。

审判还是不审判：在eBay使用大型语言模型进行广告主关键短语相关性判断