This study addresses the issue observed in Large Vision Language Models
(LVLMs), where excessive attention on a few image tokens, referred to as blind
tokens, leads to hallucinatory responses in tasks requiring fine-grained
understanding of visual objects. We found that tokens receiving lower attention
weights often hold essential information for identifying nuanced object details
-- ranging from merely recognizing object existence to identifying their
attributes (color, position, etc.) and understanding their relationships. To
counteract the over-emphasis on blind tokens and to accurately respond to user
queries, we introduce a technique called Attentional Vision Calibration (AVC).
During the decoding phase, AVC identifies blind tokens by analyzing the
image-related attention distribution. It then dynamically adjusts the logits
for the next token prediction by contrasting the logits conditioned on the
original visual tokens with those conditioned on the blind tokens. This
effectively lowers the dependency on blind tokens and promotes a more balanced
consideration of all tokens. We validate AVC on benchmarks such as POPE, MME,
and AMBER, where it consistently outperforms existing decoding techniques in
mitigating object hallucinations in LVLMs.

通过 Attentional Vision Calibration (AVC) 技术，针对 Large Vision Language Models (LVLMs) 中出现的盲目令牌（blind tokens）引发的视觉对象细节理解问题，通过动态调整逻辑回归预测以降低对盲目令牌的依赖，从而提高所有令牌的平衡考虑，有效地减少了 LVLMs 中的视觉对象幻觉。

不忘初心：大规模视觉语言模型的关注力视觉校准

Don't Miss the Forest for the Trees: Attentional Vision Calibration for  Large Vision Language Models

Large language models (LLMs) exhibit impressive natural language capabilities
but suffer from hallucination -- generating content ungrounded in the realities
of training data. Recent work has focused on decoding techniques to improve
factuality during inference by leveraging LLMs' hierarchical representation of
factual knowledge, manipulating the predicted distributions at inference time.
Current state-of-the-art approaches refine decoding by contrasting early-exit
distributions from a lower layer with the final layer to exploit information
related to factuality within the model forward procedure. However, such methods
often assume the final layer is the most reliable and the lower layer selection
process depends on it. In this work, we first propose extrapolation of critical
token probabilities beyond the last layer for more accurate contrasting. We
additionally employ layer-wise entropy-guided lower layer selection, decoupling
the selection process from the final layer. Experiments demonstrate strong
performance - surpassing state-of-the-art on multiple different datasets by
large margins. Analyses show different kinds of prompts respond to different
selection strategies.

大型语言模型 (LLMs) 具备令人印象深刻的自然语言处理能力，但存在虚构结果的问题。最近的研究关注于解码技术，通过利用 LLMs 的分层表示技术及在推理过程中操纵预测分布，提高真实性。本文通过超出最后一层的关键标记概率外推和基于层次熵的下层选择，解耦选择过程与最后一层之间的关系，实验结果显示非常好的性能，并在多个数据集上大幅领先，分析结果表明不同类型的提示需要不同的选择策略。

基于熵指导的外推解码方法提升大型语言模型中的事实准确性

Entropy Guided Extrapolative Decoding to Improve Factuality in Large  Language Models

The integration of deep learning and neuroscience has been advancing rapidly,
which has led to improvements in the analysis of brain activity and the
understanding of deep learning models from a neuroscientific perspective. The
reconstruction of visual experience from human brain activity is an area that
has particularly benefited: the use of deep learning models trained on large
amounts of natural images has greatly improved its quality, and approaches that
combine the diverse information contained in visual experiences have
proliferated rapidly in recent years. In this technical paper, by taking
advantage of the simple and generic framework that we proposed (Takagi and
Nishimoto, CVPR 2023), we examine the extent to which various additional
decoding techniques affect the performance of visual experience reconstruction.
Specifically, we combined our earlier work with the following three techniques:
using decoded text from brain activity, nonlinear optimization for structural
image reconstruction, and using decoded depth information from brain activity.
We confirmed that these techniques contributed to improving accuracy over the
baseline. We also discuss what researchers should consider when performing
visual reconstruction using deep generative models trained on large datasets.
Please check our webpage at
this https URL Code is also
available at this https URL

通过深度学习和神经科学的整合，本研究使用多种不同的解码技术对视觉体验的再建进行了实验，证明了这些技术能够大大提高基线准确率。

利用多个解码输入通过潜在扩散模型改进从人脑活动中重建视觉图像

Improving visual image reconstruction from human brain activity using  latent diffusion models via multiple decoded inputs

Technology for language generation has advanced rapidly, spurred by
advancements in pre-training large models on massive amounts of data and the
need for intelligent agents to communicate in a natural manner. While
techniques can effectively generate fluent text, they can also produce
undesirable societal biases that can have a disproportionately negative impact
on marginalized populations. Language generation presents unique challenges for
biases in terms of direct user interaction and the structure of decoding
techniques. To better understand these challenges, we present a survey on
societal biases in language generation, focusing on how data and techniques
contribute to biases and progress towards reducing biases. Motivated by a lack
of studies on biases from decoding techniques, we also conduct experiments to
quantify the effects of these techniques. By further discussing general trends
and open challenges, we call to attention promising directions for research and
the importance of fairness and inclusivity considerations for language
generation applications.

通过调查，我们着重讨论了语言生成中社会偏见的数据和技术对偏见的影响及降低偏见的进展，并进行了实验来量化解码技术的影响，提出了语言生成应用公平和包容性考虑的重要性。