In recent years, large-scale pre-trained language models (PLMs) have made
extraordinary progress in most NLP tasks. But, in the unsupervised POS tagging
task, works utilizing PLMs are few and fail to achieve state-of-the-art (SOTA)
performance. The recent SOTA performance is yielded by a Guassian HMM variant
proposed by He et al. (2018). However, as a generative model, HMM makes very
strong independence assumptions, making it very challenging to incorporate
contexualized word representations from PLMs. In this work, we for the first
time propose a neural conditional random field autoencoder (CRF-AE) model for
unsupervised POS tagging. The discriminative encoder of CRF-AE can
straightforwardly incorporate ELMo word representations. Moreover, inspired by
feature-rich HMM, we reintroduce hand-crafted features into the decoder of
CRF-AE. Finally, experiments clearly show that our model outperforms previous
state-of-the-art models by a large margin on Penn Treebank and multilingual
Universal Dependencies treebank v2.0.

本研究提出了一个神经条件随机场自编码器模型，能够直接使用 ELMo 词汇表示并同时将手工设计特征加入到解码器中，实现无监督的 POS 标注任务，并在多语言大型数据集上取得了显著优势。

将预训练语言模型与手工特征结合用于无监督词性标注

Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging

Chemical patents are an important resource for chemical information. However,
few chemical Named Entity Recognition (NER) systems have been evaluated on
patent documents, due in part to their structural and linguistic complexity. In
this paper, we explore the NER performance of a BiLSTM-CRF model utilising
pre-trained word embeddings, character-level word representations and
contextualized ELMo word representations for chemical patents. We compare word
embeddings pre-trained on biomedical and chemical patent corpora. The effect of
tokenizers optimized for the chemical domain on NER performance in chemical
patents is also explored. The results on two patent corpora show that
contextualized word representations generated from ELMo substantially improve
chemical NER performance w.r.t. the current state-of-the-art. We also show that
domain-specific resources such as word embeddings trained on chemical patents
and chemical-specific tokenizers have a positive impact on NER performance.

本文探讨了 BiLSTM-CRF 模型在利用预训练的词嵌入、字符级和上下文化 ELMo 词表示在化学专利中的命名实体识别性能，并比较了医学和化学专利语料库上预训练的词嵌入对化学专利的影响。结果表明，使用基于 ELMo 的上下文词表示法可以大幅度提高在化学命名实体识别方面的表现。我们还表明，针对化学领域的资源（如在化学专利上训练的词嵌入和特定于化学的标记器）对 NER 性能有积极影响。