CTBench is introduced as a benchmark to assess language models (LMs) in
aiding clinical study design. Given study-specific metadata, CTBench evaluates
AI models' ability to determine the baseline features of a clinical trial (CT),
which include demographic and relevant features collected at the trial's start
from all participants. These baseline features, typically presented in CT
publications (often as Table 1), are crucial for characterizing study cohorts
and validating results. Baseline features, including confounders and
covariates, are also necessary for accurate treatment effect estimation in
studies involving observational data. CTBench consists of two datasets:
"CT-Repo," containing baseline features from 1,690 clinical trials sourced from
clinicaltrials.gov, and "CT-Pub," a subset of 100 trials with more
comprehensive baseline features gathered from relevant publications. Two
LM-based evaluation methods are developed to compare the actual baseline
feature lists against LM-generated responses. "ListMatch-LM" and
"ListMatch-BERT" use GPT-4o and BERT scores (at various thresholds),
respectively, for evaluation. To establish baseline results, advanced prompt
engineering techniques using LLaMa3-70B-Instruct and GPT-4o in zero-shot and
three-shot learning settings are applied to generate potential baseline
features. The performance of GPT-4o as an evaluator is validated through
human-in-the-loop evaluations on the CT-Pub dataset, where clinical experts
confirm matches between actual and LM-generated features. The results highlight
a promising direction with significant potential for improvement, positioning
CTBench as a useful tool for advancing research on AI in CT design and
potentially enhancing the efficacy and robustness of CTs.

CTBench 是一个用于评估语言模型在辅助临床研究设计方面的基准测试，通过给定特定研究元数据，CTBench 评估人工智能模型在确定临床试验的基线特征方面的能力，包括从所有参与者开始收集的人口统计学和相关特征。

CTBench: 临床试验设计中评估语言模型能力的综合基准

CTBench: A Comprehensive Benchmark for Evaluating Language Model  Capabilities in Clinical Trial Design

Training certifiably robust neural networks is an important but challenging
task. While many algorithms for (deterministic) certified training have been
proposed, they are often evaluated on different training schedules,
certification methods, and systematically under-tuned hyperparameters, making
it difficult to compare their performance. To address this challenge, we
introduce CTBENCH, a unified library and a high-quality benchmark for certified
training that evaluates all algorithms under fair settings and systematically
tuned hyperparameters. We show that (1) almost all algorithms in CTBENCH
surpass the corresponding reported performance in literature in the magnitude
of algorithmic improvements, thus establishing new state-of-the-art, and (2)
the claimed advantage of recent algorithms drops significantly when we enhance
the outdated baselines with a fair training schedule, a fair certification
method and well-tuned hyperparameters. Based on CTBENCH, we provide new
insights into the current state of certified training and suggest future
research directions. We are confident that CTBENCH will serve as a benchmark
and testbed for future research in certified training.

CTBENCH 是一个统一的库和高质量的基准测试，用于证明训练。通过在公平设置和系统调整的超参数下评估所有算法，CTBENCH 展示了几乎所有算法超越了文献中所报道的性能，并建立了新的最先进方法，同时揭示了近期算法的优势在强化基线后明显减弱。基于 CTBENCH，我们为当前的证明训练状态提供了新的见解，并提出了未来的研究方向。我们相信 CTBENCH 将成为证明训练未来研究的基准和测试平台。