The QUILT-1M dataset is the first openly available dataset containing images
harvested from various online sources. While it provides a huge data variety,
the image quality and composition is highly heterogeneous, impacting its
utility for text-conditional image synthesis. We propose an automatic pipeline
that provides predictions of the most common impurities within the images,
e.g., visibility of narrators, desktop environment and pathology software, or
text within the image. Additionally, we propose to use semantic alignment
filtering of the image-text pairs. Our findings demonstrate that by rigorously
filtering the dataset, there is a substantial enhancement of image fidelity in
text-to-image tasks.

通过使用自动化流程和语义对齐筛选图像与文本对，我们的研究发现从 QUILT-1M 数据集中滤除常见杂质可显著提高文本到图像任务中的图像保真度。

基于模型的 QUILT-1M 病理学数据集清洗用于文本条件图像合成

Model-based Cleaning of the QUILT-1M Pathology Dataset for  Text-Conditional Image Synthesis

In recent years, vision-language models (VLMs) have shown remarkable
performance on visual reasoning tasks (e.g. attributes, location). While such
tasks measure the requisite knowledge to ground and reason over a given visual
instance, they do not, however, measure the ability of VLMs to retain and
generalize such knowledge. In this work, we evaluate their ability to acquire
"visible" physical knowledge -- the information that is easily accessible from
images of static scenes, particularly across the dimensions of object color,
size and space. We build an automatic pipeline to derive a comprehensive
knowledge resource for calibrating and probing these models. Our results
indicate a severe gap between model and human performance across all three
tasks. Furthermore, our caption pretrained baseline (CapBERT) significantly
outperforms VLMs on both size and spatial tasks -- highlighting that despite
sufficient access to ground language with visual modality, they struggle to
retain such knowledge. The dataset and code are available at
this https URL .

本研究使用自动管道方法研究了视觉 - 语言模型在获取 “可见” 物理知识方面的能力，并发现这些模型在物体颜色，大小和空间三个方面的表现与人类存在明显差距。而采用预训练的基线模型 (CapBERT) 可以取得更好的表现。

VIPHY：探究 “可见” 的物理常识知识

VIPHY: Probing "Visible" Physical Commonsense Knowledge

Tracking progress in machine learning has become increasingly difficult with
the recent explosion in the number of papers. In this paper, we present AxCell,
an automatic machine learning pipeline for extracting results from papers.
AxCell uses several novel components, including a table segmentation subtask,
to learn relevant structural knowledge that aids extraction. When compared with
existing methods, our approach significantly improves the state of the art for
results extraction. We also release a structured, annotated dataset for
training models for results extraction, and a dataset for evaluating the
performance of models on this task. Lastly, we show the viability of our
approach enables it to be used for semi-automated results extraction in
production, suggesting our improvements make this task practically viable for
the first time. Code is available on GitHub.

本文提供了一种自动机器学习流程 AxCell，通过表格分割等组件学习相关结构知识，从而改进了现有的研究结果提取方法，并提供了结构化的数据集进行模型训练和性能评估。最后，研究表明该方法的可行性，使得结果提取任务第一次在生产中半自动进行。