The conservation of stone-based cultural heritage sites is a critical concern
for preserving cultural and historical landmarks. With the advent of Large
Multimodal Models, as GPT-4omni (OpenAI), Claude 3 Opus (Anthropic) and Gemini
1.5 Pro (Google), it is becoming increasingly important to define the
operational capabilities of these models. In this work, we systematically
evaluate the abilities of the main foundational multimodal models to recognise
and classify anomalies and deterioration patterns of the stone elements that
are useful in the practice of conservation and restoration of world heritage.
After defining a taxonomy of the main stone deterioration patterns and
anomalies, we asked the foundational models to identify a curated selection of
354 highly representative images of stone-built heritage, offering them a
careful selection of labels to choose from. The result, which varies depending
on the type of pattern, allowed us to identify the strengths and weaknesses of
these models in the field of heritage conservation and restoration.

通过对主要基础多模态模型的能力进行评估，我们系统性地研究了这些模型在识别和分类有助于文化遗产保护和修复的石质元素的异常和磨损模式方面的能力，并确定了这些模型在遗产保护与修复领域的优势和劣势。

基于大型多模态模型的石材劣化模式识别

Identification of Stone Deterioration Patterns with Large Multimodal  Models

Reinforcement Learning from Human Feedback (RLHF) has been crucial to the
recent success of Large Language Models (LLMs), however, it is often a complex
and brittle process. In the classical RLHF framework, a reward model is first
trained to represent human preferences, which is in turn used by an online
reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue
with such methods is \emph{reward over-optimization} or \emph{reward hacking},
where performance as measured by the learned proxy reward model increases, but
true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs)
like Direct Preference Optimization have emerged as alternatives to the
classical RLHF pipeline by circumventing the reward modeling phase. However,
although DAAs do not use a separate proxy reward model, they still commonly
deteriorate from over-optimization. While the so-called reward hacking
phenomenon is not well-defined for DAAs, we still uncover similar trends: at
higher KL budgets, DAA algorithms exhibit similar degradation patterns to their
classic RLHF counterparts. In particular, we find that DAA methods deteriorate
not only across a wide range of KL budgets but also often before even a single
epoch of the dataset is completed. Through extensive empirical experimentation,
this work formulates and formalizes the reward over-optimization or hacking
problem for DAAs and explores its consequences across objectives, training
regimes, and model scales.

通过大量实证实验，本研究对于直接对齐算法的奖励过度优化或者篡改问题进行了形式化，并探讨了在目标、训练方式和模型规模等方面的相关影响。