We present MIPS, a novel method for program synthesis based on automated
mechanistic interpretability of neural networks trained to perform the desired
task, auto-distilling the learned algorithm into Python code. We test MIPS on a
benchmark of 62 algorithmic tasks that can be learned by an RNN and find it
highly complementary to GPT-4: MIPS solves 32 of them, including 13 that are
not solved by GPT-4 (which also solves 30). MIPS uses an integer autoencoder to
convert the RNN into a finite state machine, then applies Boolean or integer
symbolic regression to capture the learned algorithm. As opposed to large
language models, this program synthesis technique makes no use of (and is
therefore not limited by) human training data such as algorithms and code from
GitHub. We discuss opportunities and challenges for scaling up this approach to
make machine-learned models more interpretable and trustworthy.

基于神经网络的机制可解释性自动合成方法 MIPS，将学习到的算法自动转化为 Python 代码，解决了 62 个可以由 RNN 学习的算法任务中的 32 个问题，并对 GPT-4 的补充具有高度互补性，且不使用人工训练数据。

AI 黑盒子的开启：基于机械解释的程序合成

Opening the AI black box: program synthesis via mechanistic  interpretability

Retrieval finds a small number of relevant candidates from a large corpus for
information retrieval and recommendation applications. A key component of
retrieval is to model (user, item) similarity, which is commonly represented as
the dot product of two learned embeddings. This formulation permits efficient
inference, commonly known as Maximum Inner Product Search (MIPS). Despite its
popularity, dot products cannot capture complex user-item interactions, which
are multifaceted and likely high rank. We hence examine non-dot-product
retrieval settings on accelerators, and propose \textit{mixture of logits}
(MoL), which models (user, item) similarity as an adaptive composition of
elementary similarity functions. This new formulation is expressive, capable of
modeling high rank (user, item) interactions, and further generalizes to the
long tail. When combined with a hierarchical retrieval strategy,
\textit{h-indexer}, we are able to scale up MoL to 100M corpus on a single GPU
with latency comparable to MIPS baselines. On public datasets, our approach
leads to uplifts of up to 77.3\% in hit rate (HR). Experiments on a large
recommendation surface at Meta showed strong metric gains and reduced
popularity bias, validating the proposed approach's performance and improved
generalization.

通过使用混合逻辑 (MoL) 模型代替点积来准确表示复杂的用户 - 物品互动，结合	extit {h-indexer} 层级检索策略能够在单个 GPU 上扩展到 1 亿个语料库，并在公共数据集中取得了高达 77.3％的命中率提高。

重新审视加速器上的神经检索

Revisiting Neural Retrieval on Accelerators

The MIPS (maximum inner product search), which finds the item with the
highest inner product with a given query user, is an essential problem in the
recommendation field. It is usual that e-commerce companies face situations
where they want to promote and sell new or discounted items. In these
situations, we have to consider a question: who are interested in the items and
how to find them? This paper answers this question by addressing a new problem
called reverse maximum inner product search (reverse MIPS). Given a query
vector and two sets of vectors (user vectors and item vectors), the problem of
reverse MIPS finds a set of user vectors whose inner product with the query
vector is the maximum among the query and item vectors. Although the importance
of this problem is clear, its straightforward implementation incurs a
computationally expensive cost. We therefore propose Simpfer, a simple, fast,
and exact algorithm for reverse MIPS. In an offline phase, Simpfer builds a
simple index that maintains a lower-bound of the maximum inner product. By
exploiting this index, Simpfer judges whether the query vector can have the
maximum inner product or not, for a given user vector, in a constant time.
Besides, our index enables filtering user vectors, which cannot have the
maximum inner product with the query vector, in a batch. We theoretically
demonstrate that Simpfer outperforms baselines employing state-of-the-art MIPS
techniques. Furthermore, our extensive experiments on real datasets show that
Simpfer is at least two orders magnitude faster than the baselines.

本文提出了一种新的问题，称为 “反向最大内积搜索 (RMIPS)”，并通过提出一种名为 Simpfer 的简单、快速和准确的算法来解决，能够在用户向量和项目向量中找到与查询向量最大内积的向量，实验证明其性能优于 MIPS 技术并且更快。