Automated hiring systems are among the fastest-developing of all high-stakes
AI systems. Among these are algorithmic personality tests that use insights
from psychometric testing, and promise to surface personality traits indicative
of future success based on job seekers' resumes or social media profiles. We
interrogate the validity of such systems using stability of the outputs they
produce, noting that reliability is a necessary, but not a sufficient,
condition for validity. Our approach is to (a) develop a methodology for an
external audit of stability of predictions made by algorithmic personality
tests, and (b) instantiate this methodology in an audit of two systems,
Humantic AI and Crystal. Crucially, rather than challenging or affirming the
assumptions made in psychometric testing -- that personality is a meaningful
and measurable construct, and that personality traits are indicative of future
success on the job -- we frame our methodology around testing the underlying
assumptions made by the vendors of the algorithmic personality tests
themselves.
Our main contribution is the development of a socio-technical framework for
auditing the stability of algorithmic systems. This contribution is
supplemented with an open-source software library that implements the technical
components of the audit, and can be used to conduct similar stability audits of
algorithmic systems. We instantiate our framework with the audit of two
real-world personality prediction systems, namely Humantic AI and Crystal. The
application of our audit framework demonstrates that both these systems show
substantial instability with respect to key facets of measurement, and hence
cannot be considered valid testing instruments.

本篇研究针对应聘自动化招聘系统中采用心理测量学的算法性格测试的有效性，使用一种外部稳定性审计方法，通过对两种系统 Humantic AI 和 Crystal 进行测试，发现这些系统在重要方面存在实质性不稳定性，因此不能被视为有效的测试工具。

AI 招聘中使用的人格预测有效性的外部稳定性审核框架

An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring

This paper adapts topic models to the psychometric testing of MOOC students
based on their online forum postings. Measurement theory from education and
psychology provides statistical models for quantifying a person's attainment of
intangible attributes such as attitudes, abilities or intelligence. Such models
infer latent skill levels by relating them to individuals' observed responses
on a series of items such as quiz questions. The set of items can be used to
measure a latent skill if individuals' responses on them conform to a Guttman
scale. Such well-scaled items differentiate between individuals and inferred
levels span the entire range from most basic to the advanced. In practice,
education researchers manually devise items (quiz questions) while optimising
well-scaled conformance. Due to the costly nature and expert requirements of
this process, psychometric testing has found limited use in everyday teaching.
We aim to develop usable measurement models for highly-instrumented MOOC
delivery platforms, by using participation in automatically-extracted online
forum topics as items. The challenge is to formalise the Guttman scale
educational constraint and incorporate it into topic models. To favour topics
that automatically conform to a Guttman scale, we introduce a novel
regularisation into non-negative matrix factorisation-based topic modelling. We
demonstrate the suitability of our approach with both quantitative experiments
on three Coursera MOOCs, and with a qualitative survey of topic
interpretability on two MOOCs by domain expert interviews.

该研究采用话题建模来对 MOOC 学生进行心理测量学测试，并通过在线论坛发表的主题参与度来开发可用的测量模型，进而在三个 Coursera MOOC 和两个 MOOC 领域专家面试中进行质量和定量实验。