We aim at improving spoken language modeling (LM) using very large amount of
automatically transcribed speech. We leverage the INA (French National
Audiovisual Institute) collection and obtain 19GB of text after applying ASR on
350,000 hours of diverse TV shows. From this, spoken language models are
trained either by fine-tuning an existing LM (FlauBERT) or through training a
LM from scratch. New models (FlauBERT-Oral) are shared with the community and
evaluated for 3 downstream tasks: spoken language understanding, classification
of TV shows and speech syntactic parsing. Results show that FlauBERT-Oral can
be beneficial compared to its initial FlauBERT version demonstrating that,
despite its inherent noisy nature, ASR-generated text can be used to build
spoken language models.

使用自动转录的大规模语音来改进口语语言建模，利用法国国家音频视觉学院收集的文本数据应用 ASR，并在现有模型（FlauBERT）微调或从头开始训练口语语言模型，生成新模型（FlauBERT-Oral），并在口语理解、电视节目分类和语音句法分析 3 个下游任务中评估其性能，结果表明，与初始化 FlauBERT 版本相比，FlauBERT-Oral 有望带来好处，表明尽管 ASR 生成的文本存在噪音，但仍可以用于构建口语语言模型。