Disentangled representation learning remains challenging as ground truth
factors of variation do not naturally exist. To address this, we present
Vocabulary Disentanglement Retrieval~(VDR), a simple yet effective
retrieval-based disentanglement framework that leverages nature language as
distant supervision. Our approach is built upon the widely-used bi-encoder
architecture with disentanglement heads and is trained on data-text pairs that
are readily available on the web or in existing datasets. This makes our
approach task- and modality-agnostic with potential for a wide range of
downstream applications. We conduct experiments on 16 datasets in both
text-to-text and cross-modal scenarios and evaluate VDR in a zero-shot setting.
With the incorporation of disentanglement heads and a minor increase in
parameters, VDR achieves significant improvements over the base retriever it is
built upon, with a 9% higher on NDCG@10 scores in zero-shot text-to-text
retrieval and an average of 13% higher recall in cross-modal retrieval. In
comparison to other baselines, VDR outperforms them in most tasks, while also
improving explainability and efficiency.

本文提出了一种基于语言远程监督的检索式分离框架，称为 Vocabulary Disentanglement Retrieval~(VDR)，其依赖于生物编码器架构并具有解缠头，经过与其他基线方法的比较，VDR 在大多数任务中都表现优异，并提高了可解释性和效率。

基于远程监督的检索式解耦

Retrieval-based Disentanglement with Distant Supervision

Recently, the task of distantly supervised (DS) ultra-fine entity typing has
received significant attention. However, DS data is noisy and often suffers
from missing or wrong labeling issues resulting in low precision and low
recall. This paper proposes a novel ultra-fine entity typing model with
denoising capability. Specifically, we build a noise model to estimate the
unknown labeling noise distribution over input contexts and noisy type labels.
With the noise model, more trustworthy labels can be recovered by subtracting
the estimated noise from the input. Furthermore, we propose an entity typing
model, which adopts a bi-encoder architecture, is trained on the denoised data.
Finally, the noise model and entity typing model are trained iteratively to
enhance each other. We conduct extensive experiments on the Ultra-Fine entity
typing dataset as well as OntoNotes dataset and demonstrate that our approach
significantly outperforms other baseline methods.

本文提出一种新颖的超细实体类型模型，其具有去噪能力，通过建立一个噪声模型对输入的上下文和实体类型标签进行估计，从输入中减去估计噪声以恢复更可靠的标签，并采用双编码器结构训练去噪数据，噪声模型和实体类型模型迭代训练，以提高彼此的性能，在 Ultra-Fine 实体类型数据集以及 OntoNotes 数据集上表现显著优于其他基线方法。

去噪增强远程监督的超细实体类型识别

Denoising Enhanced Distantly Supervised Ultrafine Entity Typing

Given its effectiveness on knowledge-intensive natural language processing
tasks, dense retrieval models have become increasingly popular. Specifically,
the de-facto architecture for open-domain question answering uses two
isomorphic encoders that are initialized from the same pretrained model but
separately parameterized for questions and passages. This bi-encoder
architecture is parameter-inefficient in that there is no parameter sharing
between encoders. Further, recent studies show that such dense retrievers
underperform BM25 in various settings. We thus propose a new architecture,
Task-aware Specialization for dense Retrieval (TASER), which enables parameter
sharing by interleaving shared and specialized blocks in a single encoder. Our
experiments on five question answering datasets show that \ourmodel\ can
achieve superior accuracy, surpassing BM25, while using about 60% of the
parameters as bi-encoder dense retrievers. In out-of-domain evaluations, TASER
is also empirically more robust than bi-encoder dense retrievers.

提出了一种新的体系结构，Task-aware Specialization for dense Retrieval (TASER)，该模型通过交错共享模块和专用模块在单个编码器中实现参数共享，能够在使用密集检索模型时提高准确性并显著减少了参数数量，在五个问答数据集上实验表明，与双编码器密集检索器相比，TASER 能够在使用大约 60% 的参数的同时实现卓越的准确性。