Ensuring high-quality data is paramount for maximizing the performance of
machine learning models and business intelligence systems. However, challenges
in data quality, including noise in data capture, missing records, limited data
production, and confounding variables, significantly constrain the potential
performance of these systems. In this study, we propose an
architecture-agnostic algorithm, Gradient Guided Hypotheses (GGH), designed to
address these challenges. GGH analyses gradients from hypotheses as a proxy of
distinct and possibly contradictory patterns in the data. This framework
entails an additional step in machine learning training, where gradients can be
included or excluded from backpropagation. In this manner, missing and noisy
data are addressed through a unified solution that perceives both challenges as
facets of the same overarching issue: the propagation of erroneous information.
Experimental validation of GGH is conducted using real-world open-source
datasets, where records with missing rates of up to 98.5% are simulated.
Comparative analysis with state-of-the-art imputation methods demonstrates a
substantial improvement in model performance achieved by GGH. Specifically in
very high scarcity regimes, GGH was found to be the only viable solution.
Additionally, GGH's noise detection capabilities are showcased by introducing
simulated noise into the datasets and observing enhanced model performance
after filtering out the noisy data. This study presents GGH as a promising
solution for improving data quality and model performance in various
applications.

在本研究中，我们提出了一种架构无关的算法，梯度引导假设（GGH），旨在应对数据质量方面的挑战。GGH 通过分析假设的梯度作为数据中不同且可能相互矛盾的模式的代理来解决缺失和噪声数据问题，并通过机器学习训练中的额外步骤将梯度包含或排除在反向传播中。通过对真实开源数据集进行实验验证，GGH 显示出显著提高模型性能的能力，并且在极高的稀缺度情况下，GGH 是唯一可行的解决方案。此研究展示了 GGH 作为改善各种应用中的数据质量和模型性能的有希望的解决方案。