Annotation of discourse relations is a known difficult task, especially for
non-expert annotators. In this paper, we investigate novice annotators'
uncertainty on the annotation of discourse relations on spoken conversational
data. We find that dialogue context (single turn, pair of turns within speaker,
and pair of turns across speakers) is a significant predictor of confidence
scores. We compute distributed representations of discourse relations from
co-occurrence statistics that incorporate information about confidence scores
and dialogue context. We perform a hierarchical clustering analysis using these
representations and show that weighting discourse relation representations with
information about confidence and dialogue context coherently models our
annotators' uncertainty about discourse relation labels.

对话语境及刚入门的注释者的不确定性是注释语篇关系的困难任务，我们通过计算包含置信度和对话语境信息的共现统计数据中的语篇关系的分布表示，并进行层次聚类分析，发现计算这些表示并赋予关于置信度和对话语境的信息能够连贯地模拟注释者对于语篇关系标签的不确定性。