Identifying user's opinions and stances in long conversation threads on
various topics can be extremely critical for enhanced personalization, market
research, political campaigns, customer service, conflict resolution, targeted
advertising, and content moderation. Hence, training language models to
automate this task is critical. However, to train such models, gathering manual
annotations has multiple challenges: 1) It is time-consuming and costly; 2)
Conversation threads could be very long, increasing chances of noisy
annotations; and 3) Interpreting instances where a user changes their opinion
within a conversation is difficult because often such transitions are subtle
and not expressed explicitly. Inspired by the recent success of large language
models (LLMs) for complex natural language processing (NLP) tasks, we leverage
Mistral Large and GPT-4 to automate the human annotation process on the
following two tasks while also providing reasoning: i) User Stance
classification, which involves labeling a user's stance of a post in a
conversation on a five-point scale; ii) User Dogmatism classification, which
deals with labeling a user's overall opinion in the conversation on a
four-point scale. The majority voting on zero-shot, one-shot, and few-shot
annotations from these two LLMs on 764 multi-user Reddit conversations helps us
curate the USDC dataset. USDC is then used to finetune and instruction-tune
multiple deployable small language models for the 5-class stance and 4-class
dogmatism classification tasks. We make the code and dataset publicly available
[this https URL].

通过利用大型语言模型 (LLMs) 自动化人工注释过程，我们训练了多个可部署的小型语言模型来进行用户态度和教条主义的分类任务。

USDC：长对话中的用户立场和教条主义的数据集

USDC: A Dataset of $\underline{U}$ser $\underline{S}$tance and  $\underline{D}$ogmatism in Long $\underline{C}$onversations

The task of rumour verification in social media concerns assessing the
veracity of a claim on the basis of conversation threads that result from it.
While previous work has focused on predicting a veracity label, here we
reformulate the task to generate model-centric, free-text explanations of a
rumour's veracity. We follow an unsupervised approach by first utilising
post-hoc explainability methods to score the most important posts within a
thread and then we use these posts to generate informative explanatory
summaries by employing template-guided summarisation. To evaluate the
informativeness of the explanatory summaries, we exploit the few-shot learning
capabilities of a large language model (LLM). Our experiments show that LLMs
can have similar agreement to humans in evaluating summaries. Importantly, we
show that explanatory abstractive summaries are more informative and better
reflect the predicted rumour veracity than just using the highest ranking posts
in the thread.

社交媒体上的谣言验证任务涉及根据产生的会话线程来评估主张的真实性。我们通过使用事后可解释方法评分来得到最重要的帖子，然后使用这些帖子通过模板引导总结生成信息性的解释总结，表明解释的抽象总结比仅使用线程中排名最高的帖子更具信息性并更好地反映了预测的谣言真实性，从而重新制定此任务。