Large Language Models (LLMs) have recently demonstrated exceptional
performance in various Natural Language Processing (NLP) tasks. They have also
shown the ability to perform chain-of-thought (CoT) reasoning to solve complex
problems. Recent studies have explored CoT reasoning in complex multimodal
scenarios, such as the science question answering task, by fine-tuning
multimodal models with high-quality human-annotated CoT rationales. However,
collecting high-quality COT rationales is usually time-consuming and costly.
Besides, the annotated rationales are hardly accurate due to the redundant
information involved or the essential information missed. To address these
issues, we propose a novel method termed \emph{T-SciQ} that aims at teaching
science question answering with LLM signals. The T-SciQ approach generates
high-quality CoT rationales as teaching signals and is advanced to train much
smaller models to perform CoT reasoning in complex modalities. Additionally, we
introduce a novel data mixing strategy to produce more effective teaching data
samples for simple and complex science question answer problems. Extensive
experimental results show that our T-SciQ method achieves a new
state-of-the-art performance on the ScienceQA benchmark, with an accuracy of
96.18%. Moreover, our approach outperforms the most powerful fine-tuned
baseline by 4.5%.

提出了一种名为 T-SciQ 的新方法，通过使用 LLM 信号生成高质量的思维链标理，训练具有 CoT 推理能力的模型并在科学问题回答中获得了最新的最高性能。

T-SciQ: 通过大语言模型信号进行多模态思维推理教学，用于科学问答

T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large  Language Model Signals for Science Question Answering

When answering a question, humans utilize the information available across
different modalities to synthesize a consistent and complete chain of thought
(CoT). This process is normally a black box in the case of deep learning models
like large-scale language models. Recently, science question benchmarks have
been used to diagnose the multi-hop reasoning ability and interpretability of
an AI system. However, existing datasets fail to provide annotations for the
answers, or are restricted to the textual-only modality, small scales, and
limited domain diversity. To this end, we present Science Question Answering
(ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice
questions with a diverse set of science topics and annotations of their answers
with corresponding lectures and explanations. We further design language models
to learn to generate lectures and explanations as the chain of thought (CoT) to
mimic the multi-hop reasoning process when answering ScienceQA questions.
ScienceQA demonstrates the utility of CoT in language models, as CoT improves
the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in
fine-tuned UnifiedQA. We also explore the upper bound for models to leverage
explanations by feeding those in the input; we observe that it improves the
few-shot performance of GPT-3 by 18.96%. Our analysis further shows that
language models, similar to humans, benefit from explanations to learn from
fewer data and achieve the same performance with just 40% of the data. The data
and code are available at this https URL.

利用多模态多选问题和对应的讲座和解释构建科学问题回答 (ScienceQA) 基准测试，证明思路链在语言模型中的实用性，并通过在 GPT-3 上实现 18.96% 的 few-shot leap 提高上界；证明与人类类似，从解释中受益，可以借少训练数据实现相同的性能

学会阐述：基于思维链的多模态推理用于科学问答

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

We propose a novel Chain Guided Retriever-reader ({\tt CGR}) framework to
model the reasoning chain for multi-hop Science Question Answering. Our
framework is capable of performing explainable reasoning without the need of
any corpus-specific annotations, such as the ground-truth reasoning chain, or
human-annotated entity mentions. Specifically, we first generate reasoning
chains from a semantic graph constructed by Abstract Meaning Representation of
retrieved evidence facts. A \textit{Chain-aware loss}, concerning both local
and global chain information, is also designed to enable the generated chains
to serve as distant supervision signals for training the retriever, where
reinforcement learning is also adopted to maximize the utility of the reasoning
chains. Our framework allows the retriever to capture step-by-step clues of the
entire reasoning process, which is not only shown to be effective on two
challenging multi-hop Science QA tasks, namely OpenBookQA and ARC-Challenge,
but also favors explainability.

提出一种无需语料库特定注释的 Chain Guided Retriever-reader（CGR）框架，通过生成由语义图构建的推理链，提高多跳科学问答的可解释性和性能，采用增强学习训练检索器，并设计了适用于本地和全局链信息的 Chain-aware loss 作为培训的远程监督信号。