Traditional language models, adept at next-token prediction in text
sequences, often struggle with transduction tasks between distinct symbolic
systems, particularly when parallel data is scarce. Addressing this issue, we
introduce \textit{symbolic autoencoding} ($\Sigma$AE), a self-supervised
framework that harnesses the power of abundant unparallel data alongside
limited parallel data. $\Sigma$AE connects two generative models via a discrete
bottleneck layer and is optimized end-to-end by minimizing reconstruction loss
(simultaneously with supervised loss for the parallel data), such that the
sequence generated by the discrete bottleneck can be read out as the transduced
input sequence. We also develop gradient-based methods allowing for efficient
self-supervised sequence learning despite the discreteness of the bottleneck.
Our results demonstrate that $\Sigma$AE significantly enhances performance on
transduction tasks, even with minimal parallel data, offering a promising
solution for weakly supervised learning scenarios.

通过最小化重建损失以及监督损失，符号自编码器（ΣAE）利用大量非平行数据和有限的平行数据，通过离散瓶颈层将两个生成模型连接起来，从而在传递任务中显着提高性能，并为弱监督学习场景提供了有希望的解决方案。