This paper presents a novel speaking-style captioning method that generates
diverse descriptions while accurately predicting speaking-style information.
Conventional learning criteria directly use original captions that contain not
only speaking-style factor terms but also syntax words, which disturbs learning
speaking-style information. To solve this problem, we introduce
factor-conditioned captioning (FCC), which first outputs a phrase representing
speaking-style factors (e.g., gender, pitch, etc.), and then generates a
caption to ensure the model explicitly learns speaking-style factors. We also
propose greedy-then-sampling (GtS) decoding, which first predicts
speaking-style factors deterministically to guarantee semantic accuracy, and
then generates a caption based on factor-conditioned sampling to ensure
diversity. Experiments show that FCC outperforms the original caption-based
training, and with GtS, it generates more diverse captions while keeping style
prediction performance.

本篇论文提出了一种新颖的说话风格字幕生成方法，能够生成多样的描述，同时准确地预测说话风格信息。通过引入因素条件字幕生成（FCC）和贪婪采样（GtS）解码方法，实现对说话风格因素的学习和生成多样化的字幕，提高了风格预测性能。

因素条件下的言语风格字幕生成

Factor-Conditioned Speaking-Style Captioning

People convey their intention and attitude through linguistic styles of the
text that they write. In this study, we investigate lexicon usages across
styles throughout two lenses: human perception and machine word importance,
since words differ in the strength of the stylistic cues that they provide. To
collect labels of human perception, we curate a new dataset, Hummingbird, on
top of benchmarking style datasets. We have crowd workers highlight the
representative words in the text that makes them think the text has the
following styles: politeness, sentiment, offensiveness, and five emotion types.
We then compare these human word labels with word importance derived from a
popular fine-tuned style classifier like BERT. Our results show that the BERT
often finds content words not relevant to the target style as important words
used in style prediction, but humans do not perceive the same way even though
for some styles (e.g., positive sentiment and joy) human- and
machine-identified words share significant overlap for some styles.

本研究通过探究人类感知和机器词汇重要性这两个视角，研究了文本的语言风格对词汇用法的影响，通过收集人类感知数据 Hummingbird 并参考目前常用的 BERT 语言风格分类器，比较了两者对于词汇标签的不同理解。研究表明，对于某些风格，如积极情感和愉悦，人类和机器的识别结果存在显著的重叠，但机器标记某些内容词非风格相关，而人们不能通过这些词汇准确感知对应的风格。