Existing industrial anomaly detection (IAD) methods predict anomaly scores
for both anomaly detection and localization. However, they struggle to perform
a multi-turn dialog and detailed descriptions for anomaly regions, e.g., color,
shape, and categories of industrial anomalies. Recently, large multimodal
(i.e., vision and language) models (LMMs) have shown eminent perception
abilities on multiple vision tasks such as image captioning, visual
understanding, visual reasoning, etc., making it a competitive potential choice
for more comprehensible anomaly detection. However, the knowledge about anomaly
detection is absent in existing general LMMs, while training a specific LMM for
anomaly detection requires a tremendous amount of annotated data and massive
computation resources. In this paper, we propose a novel large multi-modal
model by applying vision experts for industrial anomaly detection (dubbed
Myriad), which leads to definite anomaly detection and high-quality anomaly
description. Specifically, we adopt MiniGPT-4 as the base LMM and design an
Expert Perception module to embed the prior knowledge from vision experts as
tokens which are intelligible to Large Language Models (LLMs). To compensate
for the errors and confusions of vision experts, we introduce a domain adapter
to bridge the visual representation gaps between generic and industrial images.
Furthermore, we propose a Vision Expert Instructor, which enables the Q-Former
to generate IAD domain vision-language tokens according to vision expert prior.
Extensive experiments on MVTec-AD and VisA benchmarks demonstrate that our
proposed method not only performs favorably against state-of-the-art methods
under the 1-class and few-shot settings, but also provide definite anomaly
prediction along with detailed descriptions in IAD domain.

提出了一种应用视觉专家进行工业异常检测的新型大型多模型（Myriad）模型，通过使用视觉专家的先验知识嵌入到大型语言模型中，实现了明确的异常检测和高质量的异常描述。

Myriad: 應用視覺專家進行工業異常檢測之大型多模態模型

Myriad: Large Multimodal Model by Applying Vision Experts for Industrial  Anomaly Detection

We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of
vision experts to achieve multimodal reasoning and action. In this paper, we
define and explore a comprehensive list of advanced vision tasks that are
intriguing to solve, but may exceed the capabilities of existing vision and
vision-language models. To achieve such advanced visual intelligence, MM-REACT
introduces a textual prompt design that can represent text descriptions,
textualized spatial coordinates, and aligned file names for dense visual
signals such as images and videos. MM-REACT's prompt design allows language
models to accept, associate, and process multimodal information, thereby
facilitating the synergetic combination of ChatGPT and various vision experts.
Zero-shot experiments demonstrate MM-REACT's effectiveness in addressing the
specified capabilities of interests and its wide application in different
scenarios that require advanced visual understanding. Furthermore, we discuss
and compare MM-REACT's system paradigm with an alternative approach that
extends language models for multimodal scenarios through joint finetuning.
Code, demo, video, and visualization are available at
this https URL

MM-REACT 是一种系统范例，将 ChatGPT 与一组视觉专家集成在一起，以实现多模态推理和动作，并引入了文本提示设计，以表示文本描述、已文本化的空间坐标和对齐的文件名来处理多模态信息。