The advent of large vision-language models (LVLMs) represents a noteworthy
advancement towards the pursuit of artificial general intelligence. However,
the extent of their efficacy across both specialized and general tasks warrants
further investigation. This article endeavors to evaluate the competency of
popular LVLMs in specialized and general tasks, respectively, aiming to offer a
comprehensive comprehension of these innovative methodologies. To gauge their
efficacy in specialized tasks, we tailor a comprehensive testbed comprising
three distinct scenarios: natural, healthcare, and industrial, encompassing six
challenging tasks. These tasks include salient, camouflaged, and transparent
object detection, as well as polyp and skin lesion detection, alongside
industrial anomaly detection. We examine the performance of three recent
open-source LVLMs -- MiniGPT-v2, LLaVA-1.5, and Shikra -- in the realm of
visual recognition and localization. Moreover, we conduct empirical
investigations utilizing the aforementioned models alongside GPT-4V, assessing
their multi-modal understanding capacities in general tasks such as object
counting, absurd question answering, affordance reasoning, attribute
recognition, and spatial relation reasoning. Our investigations reveal that
these models demonstrate limited proficiency not only in specialized tasks but
also in general tasks. We delve deeper into this inadequacy and suggest several
potential factors, including limited cognition in specialized tasks, object
hallucination, text-to-image interference, and decreased robustness in complex
problems. We hope this study would provide valuable insights for the future
development of LVLMs, augmenting their power in coping with both general and
specialized applications.

大视觉语言模型在专业任务和通用任务中的效能进行综合评估，发现它们在专业任务和通用任务中均表现有限，可能的因素包括专业任务认知有限、物体幻觉、文本与图像的干扰以及在复杂问题中的鲁棒性降低。

近期大型视觉 - 语言模型的有效性评估

Effectiveness Assessment of Recent Large Vision-Language Models

This paper introduces a multifaceted methodology for fine-tuning and
evaluating large language models (LLMs) for specialized monetization tasks. The
goal is to balance general language proficiency with domain-specific skills.
The methodology has three main components: 1) Carefully blending in-domain and
general-purpose data during fine-tuning to achieve an optimal balance between
general and specialized capabilities; 2) Designing a comprehensive evaluation
framework with 45 questions tailored to assess performance on functionally
relevant dimensions like reliability, consistency, and business impact; 3)
Analyzing how model size and continual training influence metrics to guide
efficient resource allocation during fine-tuning. The paper details the design,
data collection, analytical techniques, and results validating the proposed
frameworks. It aims to provide businesses and researchers with actionable
insights on effectively adapting LLMs for specialized contexts. We also intend
to make public the comprehensive evaluation framework, which includes the 45
tailored questions and their respective scoring guidelines, to foster
transparency and collaboration in adapting LLMs for specialized tasks.

介绍了用于微调和评估大型语言模型（LLMs）用于专门的货币化任务的多方面方法论，目标是在一般语言能力和领域特定技能之间实现平衡。方法论有三个主要组成部分：1）在微调过程中精心混合领域内和通用数据，以实现一般能力和专业能力之间的最佳平衡；2）设计全面的评估框架，包含 45 个问题，旨在评估在功能相关维度（如可靠性、一致性和商业影响）上的表现；3）分析模型大小和持续训练对度量指标的影响，以指导在微调过程中的高效资源分配。本文详细介绍了所提出框架的设计、数据收集、分析技术和验证结果，旨在为企业和研究人员提供行动洞察，以有效地使 LLMs 适应专门的环境。我们还打算公开全面的评估框架，其中包括 45 个量身定制的问题及其相应的评分指南，以促进 LLMs 在专门任务上的透明度与合作。

在 LLMs 中平衡专业和通用技能：现代调谐和数据策略的影响

Balancing Specialized and General Skills in LLMs: The Impact of Modern  Tuning and Data Strategy

Large foundation models have demonstrated a great ability to achieve general
human-level intelligence far beyond traditional approaches. As the technique
keeps attracting attention from the AI community, more and more large
foundation models have become publically available. However, most of those
models exhibit a major deficiency in specialized-task applications, where the
step of finetuning is still required for obtaining satisfactory performance. As
the number of available models and specialized tasks keeps growing, the job of
general finetuning becomes highly nontrivial. In this paper, we take the first
step to address this issue. We introduce an extensible and lightweight toolkit,
LMFlow, which aims to simplify the finetuning and inference of general large
foundation models. LMFlow offers a complete finetuning workflow for a large
foundation model to support personalized training with limited computing
resources. Furthermore, it supports continuous pretraining, instruction tuning,
parameter-efficient finetuning, alignment tuning, and large model inference,
along with carefully designed and extensible APIs. This toolkit has been
thoroughly tested and is available at this https URL

本文介绍了一个工具包 LMFlow，其目的是简化大型基础模型的微调和推理，并支持个性化训练和连续预训练以及大模型推理。