The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA
encounter limitations in domain-specific tasks, with these models often lacking
depth and accuracy in specialized areas, and exhibiting a decrease in general
capabilities when fine-tuned, particularly analysis ability in small sized
models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement
Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization
(PPO), demonstrating remarkable ability in in-domain scenarios without
compromising general task performance. Our exploration of ICE-GRT highlights
its understanding and reasoning ability to not only generate robust answers but
also to provide detailed analyses of the reasons behind the answer. This
capability marks a significant progression beyond the scope of Supervised
Fine-Tuning models. The success of ICE-GRT is dependent on several crucial
factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage
Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in
domain-specific tasks and across 12 general Language tasks against equivalent
size and even larger size LLMs, highlighting the effectiveness of our approach.
We provide a comprehensive analysis of the ICE-GRT, underscoring the
significant advancements it brings to the field of LLM.

利用人类反馈的强化学习从 PPO 出发，ICE-GRT 在特定领域任务中展示了出色的能力，同时保持了通用任务性能，在小型模型中表现了分析能力的下降。