End-to-end (E2E) models are often being accompanied by language models (LMs)
via shallow fusion for boosting their overall quality as well as recognition of
rare words. At the same time, several prior works show that LMs are susceptible
to unintentionally memorizing rare or unique sequ