In the domain of scientific imaging, interpreting visual data often demands
an intricate combination of human expertise and deep comprehension of the
subject materials. This study presents a novel methodology to linguistically
emulate and subsequently evaluate human-like interactions with Scanning
Electron Microscopy (SEM) images, specifically of glass materials. Leveraging a
multimodal deep learning framework, our approach distills insights from both
textual and visual data harvested from peer-reviewed articles, further
augmented by the capabilities of GPT-4 for refined data synthesis and
evaluation. Despite inherent challenges--such as nuanced interpretations and
the limited availability of specialized datasets--our model (GlassLLaVA) excels
in crafting accurate interpretations, identifying key features, and detecting
defects in previously unseen SEM images. Moreover, we introduce versatile
evaluation metrics, suitable for an array of scientific imaging applications,
which allows for benchmarking against research-grounded answers. Benefiting
from the robustness of contemporary Large Language Models, our model adeptly
aligns with insights from research papers. This advancement not only
underscores considerable progress in bridging the gap between human and machine
interpretation in scientific imaging, but also hints at expansive avenues for
future research and broader application.

该研究提出了一种新的方法，通过多模态深度学习框架从文本和视觉数据中提取洞察力，以模拟和评估人类对扫描电子显微镜 (SEM) 图像（特别是玻璃材料）的交互。我们的模型（GlassLLaVA）在解释、特征识别和检测以前未见的 SEM 图像中的缺陷方面表现出色，为科学成像应用引入了多用途的评估指标，进一步缩小人与机器解释在科学成像中的差距，为未来研究和广泛应用提供了广阔的可能性。

科学图像解读的多模态深度学习

Multimodal Deep Learning for Scientific Imaging Interpretation

Most camera lens systems are designed in isolation, separately from
downstream computer vision methods. Recently, joint optimization approaches
that design lenses alongside other components of the image acquisition and
processing pipeline -- notably, downstream neural networks -- have achieved
improved imaging quality or better performance on vision tasks. However, these
existing methods optimize only a subset of lens parameters and cannot optimize
glass materials given their categorical nature. In this work, we develop a
differentiable spherical lens simulation model that accurately captures
geometrical aberrations. We propose an optimization strategy to address the
challenges of lens design -- notorious for non-convex loss function landscapes
and many manufacturing constraints -- that are exacerbated in joint
optimization tasks. Specifically, we introduce quantized continuous glass
variables to facilitate the optimization and selection of glass materials in an
end-to-end design context, and couple this with carefully designed constraints
to support manufacturability. In automotive object detection, we report
improved detection performance over existing designs even when simplifying
designs to two- or three-element lenses, despite significantly degrading the
image quality.

通过可微分的球形镜头仿真模型和量化连续玻璃变量，结合精心设计的制造限制，提出了一种优化策略来设计具有改进成像质量的镜头，以支持终端到端的设计环境，实现对焦平面险些损失不凸的损失函数景观的优化，论文中的实验结果表明，即使简化为二或三元镜头，也能实现改进的检测性能，同时显著降低了图像质量。