In recent years, large-scale pre-trained language models (PLMs) have made
extraordinary progress in most NLP tasks. But, in the unsupervised POS tagging
task, works utilizing PLMs are few and fail to achieve state-of-the-art (SOTA)
performance. The recent SOTA performance is yielded by a Guassian HMM variant
proposed by He et al. (2018). However, as a generative model, HMM makes very
strong independence assumptions, making it very challenging to incorporate
contexualized word representations from PLMs. In this work, we for the first
time propose a neural conditional random field autoencoder (CRF-AE) model for
unsupervised POS tagging. The discriminative encoder of CRF-AE can
straightforwardly incorporate ELMo word representations. Moreover, inspired by
feature-rich HMM, we reintroduce hand-crafted features into the decoder of
CRF-AE. Finally, experiments clearly show that our model outperforms previous
state-of-the-art models by a large margin on Penn Treebank and multilingual
Universal Dependencies treebank v2.0.

本研究提出了一个神经条件随机场自编码器模型，能够直接使用 ELMo 词汇表示并同时将手工设计特征加入到解码器中，实现无监督的 POS 标注任务，并在多语言大型数据集上取得了显著优势。

将预训练语言模型与手工特征结合用于无监督词性标注

Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging

Unsupervised part of speech (POS) tagging is often framed as a clustering
problem, but practical taggers need to \textit{ground} their clusters as well.
Grounding generally requires reference labeled data, a luxury a low-resource
language might not have. In this work, we describe an approach for low-resource
unsupervised POS tagging that yields fully grounded output and requires no
labeled training data. We find the classic method of Brown et al. (1992)
clusters well in our use case and employ a decipherment-based approach to
grounding. This approach presumes a sequence of cluster IDs is a `ciphertext'
and seeks a POS tag-to-cluster ID mapping that will reveal the POS sequence. We
show intrinsically that, despite the difficulty of the task, we obtain
reasonable performance across a variety of languages. We also show
extrinsically that incorporating our POS tagger into a name tagger leads to
state-of-the-art tagging performance in Sinhalese and Kinyarwanda, two
languages with nearly no labeled POS data available. We further demonstrate our
tagger's utility by incorporating it into a true `zero-resource' variant of the
Malopa (Ammar et al., 2016) dependency parser model that removes the current
reliance on multilingual resources and gold POS tags for new languages.
Experiments show that including our tagger makes up much of the accuracy lost
when gold POS tags are unavailable.

本文描述了一种针对低资源语言的无监督词性标注方法，利用了 Brown 等人在 1992 年提出的经典聚类方法和一种解密方法实现了标注结果的有效构建，并将其应用于 Sinhalese 和 Kinyarwanda 两种极度缺乏标记训练数据的语言中，实现了较高的标注性能。此外，该方法还将集成到真正的零资源模型中，取得了可靠的效果。