The Learning-to-match (LTM) framework proves to be an effective inverse
optimal transport approach for learning the underlying ground metric between
two sources of data, facilitating subsequent matching. However, the
conventional LTM framework faces scalability challenges, necessitating the use
of the entire dataset each time the parameters of the ground metric are
updated. In adapting LTM to the deep learning context, we introduce the
mini-batch Learning-to-match (m-LTM) framework for audio-text retrieval
problems. This framework leverages mini-batch subsampling and
Mahalanobis-enhanced family of ground metrics. Moreover, to cope with
misaligned training data in practice, we propose a variant using partial
optimal transport to mitigate the harm of misaligned data pairs in training
data. We conduct extensive experiments on audio-text matching problems using
three datasets: AudioCaps, Clotho, and ESC-50. Results demonstrate that our
proposed method is capable of learning rich and expressive joint embedding
space, which achieves SOTA performance. Beyond this, the proposed m-LTM
framework is able to close the modality gap across audio and text embedding,
which surpasses both triplet and contrastive loss in the zero-shot sound event
detection task on the ESC-50 dataset. Notably, our strategy of employing
partial optimal transport with m-LTM demonstrates greater noise tolerance than
contrastive loss, especially under varying noise ratios in training data on the
AudioCaps dataset. Our code is available at
this https URL

提出了一种基于小批量学习匹配 (m-LTM) 框架的音频 - 文本检索方法，包括使用小批量子采样和马氏增强的地面度量系列以及使用部分最优传输来应对训练数据中的错位数据对。在多个数据集上的实验证明，该方法能够学习丰富、表达力强的联合嵌入空间，并取得了 SOTA 性能，同时能够弥补音频和文本嵌入之间的模态差距，在 ESC-50 数据集的零样本声音事件检测任务中超越了三元组和对比损失方法。值得注意的是，在音频数据集 AudioCaps 上，在训练数据中的噪声比例变化时，采用部分最优传输的 m-LTM 策略表现出比对比损失更大的噪声容忍度。

以交通视角再审视深度语音文本检索

Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation

This paper studies the Partial Optimal Transport (POT) problem between two
unbalanced measures with at most $n$ supports and its applications in various
AI tasks such as color transfer or domain adaptation. There is hence the need
for fast approximations of POT with increasingly large problem sizes in arising
applications. We first theoretically and experimentally investigate the
infeasibility of the state-of-the-art Sinkhorn algorithm for POT due to its
incompatible rounding procedure, which consequently degrades its qualitative
performance in real world applications like point-cloud registration. To this
end, we propose a novel rounding algorithm for POT, and then provide a feasible
Sinkhorn procedure with a revised computation complexity of
$\mathcal{\widetilde O}(n^2/\varepsilon^4)$. Our rounding algorithm also
permits the development of two first-order methods to approximate the POT
problem. The first algorithm, Adaptive Primal-Dual Accelerated Gradient Descent
(APDAGD), finds an $\varepsilon$-approximate solution to the POT problem in
$\mathcal{\widetilde O}(n^{2.5}/\varepsilon)$, which is better in $\varepsilon$
than revised Sinkhorn. The second method, Dual Extrapolation, achieves the
computation complexity of $\mathcal{\widetilde O}(n^2/\varepsilon)$, thereby
being the best in the literature. We further demonstrate the flexibility of POT
compared to standard OT as well as the practicality of our algorithms on real
applications where two marginal distributions are unbalanced.

本文研究了两个非平衡度量之间的部分最优输运（Partial Optimal Transport，POT）问题及其在颜色转移或领域适应等各种人工智能任务中的应用。我们首先通过理论和实验证明了现有的 Sinkhorn 算法在 POT 问题上的不可行性，进而提出了一种新的 POT 算法来解决这一问题，并且提供了几种近似 POT 问题的一阶方法，其中包括了近似解在 ε 范围内的 Adaptive Primal-Dual Accelerated Gradient Descent（APDAGD）算法以及具有最佳计算复杂度的 Dual Extrapolation 算法。同时，我们还展示了 POT 相比标准 OT 的灵活性以及我们算法在两个非平衡边缘分布的实际应用中的实用性。

关于部分最优输运：修正 Sinkhorn 和高效梯度方法的不可行性

On Partial Optimal Transport: Revising the Infeasibility of Sinkhorn and  Efficient Gradient Methods

In order to reduce domain discrepancy to improve the performance of
cross-domain spoken language identification (SLID) system, as an unsupervised
domain adaptation (UDA) method, we have proposed a joint distribution alignment
(JDA) model based on optimal transport (OT). A discrepancy measurement based on
OT was adopted for JDA between training and test data sets. In our previous
study, it was supposed that the training and test sets share the same label
space. However, in real applications, the label space of the test set is only a
subset of that of the training set. Fully matching training and test domains
for distribution alignment may introduce negative domain transfer. In this
paper, we propose an JDA model based on partial optimal transport (POT), i.e.,
only partial couplings of OT are allowed during JDA. Moreover, since the label
of test data is unknown, in the POT, a soft weighting on the coupling based on
transport cost is adaptively set during domain alignment. Experiments were
carried out on a cross-domain SLID task to evaluate the proposed UDA. Results
showed that our proposed UDA significantly improved the performance due to the
consideration of the partial couplings in OT.

本论文提出了基于部分最优传输的联合分布对齐模型（POT），以降低不同领域之间的差异，提高跨域口语识别（SLID）系统的性能，并在 SLID 任务上进行了实验证明了 POT 模型显著提高了模型性能。

部分耦合的最优传输在口语认别中的应用

Partial Coupling of Optimal Transport for Spoken Language Identification

Mini-batch optimal transport (m-OT) has been widely used recently to deal
with the memory issue of OT in large-scale applications. Despite their
practicality, m-OT suffers from misspecified mappings, namely, mappings that
are optimal on the mini-batch level but are partially wrong in the comparison
with the optimal transportation plan between the original measures. Motivated
by the misspecified mappings issue, we propose a novel mini-batch method by
using partial optimal transport (POT) between mini-batch empirical measures,
which we refer to as mini-batch partial optimal transport (m-POT). Leveraging
the insight from the partial transportation, we explain the source of
misspecified mappings from the m-OT and motivate why limiting the amount of
transported masses among mini-batches via POT can alleviate the incorrect
mappings. Finally, we carry out extensive experiments on various applications
such as deep domain adaptation, partial domain adaptation, deep generative
model, color transfer, and gradient flow to demonstrate the favorable
performance of m-POT compared to current mini-batch methods.

本文提出了一种基于 Partial Optimal Transport (POT) 的 Mini-batch Partial Optimal Transport (m-POT) 方法，用于解决 Mini-batch Optimal Transport (m-OT) 在大规模应用中因 misspecified mappings 引起的问题，并在各种应用中进行了广泛的实验以证明其性能优于当前的 Mini-batch 方法。