The potential of Vision-Language Models (\textsc{vlm}s) often remains
underutilized in handling complex text-based problems, particularly when these
problems could benefit from visual representation. Resonating with humans'
ability to solve complex text-based problems by (1) creating a visual diagram
from the problem and (2) deducing what steps they need to take to solve it, we
propose \textsc{Self-Imagine}. We leverage a single Vision-Language Model
(\textsc{vlm}) to generate a structured representation of the question using
HTML, then render the HTML as an image, and finally use the same \vlm to answer
the question using both the question and the image. Our approach does not
require any additional training data or training. We evaluate our approach in
three mathematics tasks and nine general-purpose reasoning tasks using
state-of-the-art \textsc{vlm}. Our approach boosts the performance of
\textsc{vlm} on all math tasks (\gsm: +4.62\%; \asdiv: +4.49\%; \svamp:
+9.30\%) and the majority of the general-purpose reasoning tasks by 0.4\% to
13.20\% while achieving comparable performance in other tasks.
Code and data at this https URL .

使用 Vision-Language Models 和 HTML 生成结构化问题的图像化表示，并使用相同的 Vision-Language Model 回答问题。该方法在数学任务和推理任务中提高了性能。

自我想象：使用多模态模型进行高效单模态推理

Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using  Self-Imagination

We introduce HTLM, a hyper-text language model trained on a large-scale web
crawl. Modeling hyper-text has a number of advantages: (1) it is easily
gathered at scale, (2) it provides rich document-level and end-task-adjacent
supervision (e.g. class and id attributes often encode document category
information), and (3) it allows for new structured prompting that follows the
established semantics of HTML (e.g. to do zero-shot summarization by infilling
title tags for a webpage that contains the input text). We show that
pretraining with a BART-style denoising loss directly on simplified HTML
provides highly effective transfer for a wide range of end tasks and
supervision levels. HTLM matches or exceeds the performance of comparably sized
text-only LMs for zero-shot prompting and fine-tuning for classification
benchmarks, while also setting new state-of-the-art performance levels for
zero-shot summarization. We also find that hyper-text prompts provide more
value to HTLM, in terms of data efficiency, than plain text prompts do for
existing LMs, and that HTLM is highly effective at auto-prompting itself, by
simply generating the most likely hyper-text formatting for any available
training data. We will release all code and models to support future HTLM
research.

介绍了一种基于 HTML 训练的超文本语言模型 HTLM，使用 BART-style denoising loss 在简化 HTML 上进行预训练可对广泛端任务和监督级别提供高效的迁移，同时比纯文本模型表现更好，能够自动完成大部分的超文本格式转换，且零样本摘要效果最佳。

HTLM: 语言模型超文本预训练和提示

HTLM: Hyper-Text Pre-Training and Prompting of Language Models

Extracting structured data from HTML documents is a long-studied problem with
a broad range of applications like augmenting knowledge bases, supporting
faceted search, and providing domain-specific experiences for key verticals
like shopping and movies. Previous approaches have either required a small
number of examples for each target site or relied on carefully handcrafted
heuristics built over visual renderings of websites. In this paper, we present
a novel two-stage neural approach, named FreeDOM, which overcomes both these
limitations. The first stage learns a representation for each DOM node in the
page by combining both the text and markup information. The second stage
captures longer range distance and semantic relatedness using a relational
neural network. By combining these stages, FreeDOM is able to generalize to
unseen sites after training on a small number of seed sites from that vertical
without requiring expensive hand-crafted features over visual renderings of the
page. Through experiments on a public dataset with 8 different verticals, we
show that FreeDOM beats the previous state of the art by nearly 3.7 F1 points
on average without requiring features over rendered pages or expensive
hand-crafted features.

本文介绍了一种名为 FreeDOM 的新型两阶段神经方法，该方法通过结合文本和标记信息以及使用关系神经网络来捕捉更长的距离和语义关联性，可以从少量的种子站点的训练中泛化到未知的站点，而无需昂贵的手工制作特征或页面的可视化呈现。实验结果表明，FreeDOM 平均比现有技术高出近 3.7 个 F1 点。