In legal decisions, split votes (SV) occur when judges cannot reach a
unanimous decision, posing a difficulty for lawyers who must navigate diverse
legal arguments and opinions. In high-stakes domains, understanding the
alignment of perceived difficulty between humans and AI systems is crucial to
build trust. However, existing NLP calibration methods focus on a classifier's
awareness of predictive performance, measured against the human majority class,
overlooking inherent human label variation (HLV). This paper explores split
votes as naturally observable human disagreement and value pluralism. We
collect judges' vote distributions from the European Court of Human Rights
(ECHR), and present SV-ECHR, a case outcome classification (COC) dataset with
SV information. We build a taxonomy of disagreement with SV-specific
subcategories. We further assess the alignment of perceived difficulty between
models and humans, as well as confidence- and human-calibration of COC models.
We observe limited alignment with the judge vote distribution. To our
knowledge, this is the first systematic exploration of calibration to human
judgements in legal NLP. Our study underscores the necessity for further
research on measuring and enhancing model calibration considering HLV in legal
decision tasks.

法律决策中的分裂投票在处理多样化的法律论点和意见时给律师带来困难。了解人类与 AI 系统之间知觉难度的一致性对于建立信任至关重要。然而，现有的 NLP 校准方法重点关注分类器对预测性能的意识，忽视了人类标签变化的固有性。本文探讨了分裂投票作为自然可观察的人类分歧和价值多元主义现象。我们从欧洲人权法院 (ECHR) 收集法官的投票分布，并提出了 SV-ECHR，一个带有分裂投票信息的案件结果分类数据集。我们建立了与分裂投票相关的异议分类体系，并进一步评估了模型和人类之间知觉难度的一致性，以及案件结果分类模型的置信度和人类标定。我们观察到模型与法官投票分布的一致性有限。据我们所知，这是首次系统探索在法律 NLP 中校准人类判断的研究。我们的研究强调了在法律决策任务中测量和提升模型校准，考虑人类标签变化的必要性。

透过分裂投票的镜头：探索法律案件结果分类中的分歧、难度和校准

Through the Lens of Split Vote: Exploring Disagreement, Difficulty and  Calibration in Legal Case Outcome Classification

Summarizing legal decisions requires the expertise of law practitioners,
which is both time- and cost-intensive. This paper presents techniques for
extractive summarization of legal decisions in a low-resource setting using
limited expert annotated data. We test a set of models that locate relevant
content using a sequential model and tackle redundancy by leveraging maximal
marginal relevance to compose summaries. We also demonstrate an implicit
approach to help train our proposed models generate more informative summaries.
Our multi-task learning model variant leverages rhetorical role identification
as an auxiliary task to further improve the summarizer. We perform extensive
experiments on datasets containing legal decisions from the US Board of
Veterans' Appeals and conduct quantitative and expert-ranked evaluations of our
models. Our results show that the proposed approaches can achieve ROUGE scores
vis-à-vis expert extracted summaries that match those achieved by
inter-annotator comparison.

本研究提出了一种在法律决策摘要中提取摘要的技术，该技术利用有限的专家注释数据在低资源环境下操作。我们测试了一组使用顺序模型定位相关内容的模型，并利用最大边际相关性来处理冗余以组合摘要。我们还展示了一种隐式方法来帮助培训我们的建议模型生成更多有信息量的摘要。我们的多任务学习模型变体通过将修辞角色识别作为辅助任务来进一步提高摘要器的性能。我们对包含来自美国退伍军人委员会的法律裁决的数据集进行了广泛的实验，并对我们的模型进行了定量和专家排名评估。我们的结果表明，所提出的方法可以实现 ROUGE 得分与与专业提取的摘要相匹配的得分。