Guidance on how to validate computational text-based measures of social
science constructs is fragmented. Whereas scholars are generally acknowledging
the importance of validating their text-based measures, they often lack common
terminology and a unified framework to do so. This paper introduces a new
validation framework called ValiTex, designed to assist scholars to measure
social science constructs based on textual data. The framework draws on a
long-established tradition within psychometrics while extending the framework
for the purpose of computational text analysis. ValiTex consists of two
components, a conceptual model, and a dynamic checklist. Whereas the conceptual
model provides a general structure along distinct phases on how to approach
validation, the dynamic checklist defines specific validation steps and
provides guidance on which steps might be considered recommendable (i.e.,
providing relevant and necessary validation evidence) or optional (i.e., useful
for providing additional supporting validation evidence. The utility of the
framework is demonstrated by applying it to a use case of detecting sexism from
social media data.

本研究提出了一种名为 ValiTex 的新的验证框架，旨在帮助学者基于文本数据对社会科学构造进行测量，并通过将心理测量学的框架扩展到计算文本分析领域，提供通用的结构和具体的验证步骤。该框架在社交媒体数据中检测性别歧视的案例中展示了其实用性。