Accurate and transparent financial information disclosure is crucial in the
fields of accounting and finance, ensuring market efficiency and investor
confidence. Among many information disclosure platforms, the Chinese stock
exchanges' investor interactive platform provides a novel and interactive way
for listed firms to disclose information of interest to investors through an
online question-and-answer (Q&A) format. However, it is common for listed firms
to respond to questions with limited or no substantive information, and
automatically evaluating the quality of financial information disclosure on
large amounts of Q&A pairs is challenging. This paper builds a benchmark
FinTruthQA, that can evaluate advanced natural language processing (NLP)
techniques for the automatic quality assessment of information disclosure in
financial Q&A data. FinTruthQA comprises 6,000 real-world financial Q&A entries
and each Q&A was manually annotated based on four conceptual dimensions of
accounting. We benchmarked various NLP techniques on FinTruthQA, including
statistical machine learning models, pre-trained language model and their
fine-tuned versions, as well as the large language model GPT-4. Experiments
showed that existing NLP models have strong predictive ability for real
question identification and question relevance tasks, but are suboptimal for
answer relevance and answer readability tasks. By establishing this benchmark,
we provide a robust foundation for the automatic evaluation of information
disclosure, significantly enhancing the transparency and quality of financial
reporting. FinTruthQA can be used by auditors, regulators, and financial
analysts for real-time monitoring and data-driven decision-making, as well as
by researchers for advanced studies in accounting and finance, ultimately
fostering greater trust and efficiency in the financial markets.

建立了一个名为 FinTruthQA 的基准，用于自动评估金融问答数据中信息披露质量，为会计和金融领域的技术研究和数据驱动的决策提供了坚实的基础。