Synthetic data for advanced analytics, AI and machine learning

Data for AI training is often hard to come by, especially when organizations use third party AI and machine learning solutions. Only 15-20% of customers consent to using their data for analytics, the rest of the data and the insights contained are locked away. Due to privacy reasons, sensitive data is often off-limits both for in-house data science teams and for external AI or analytics vendors.

Even when data is available, data quality is an issue. Historic biases and model drift complicate AI/ML development and negatively impact performance. Machine learning accuracy suffers when training data quality is insufficient. This is due to imbalanced training data. Recalibrating models without easy access to fresh, balanced training data is impossible. Models need to be able pick up on rare or completely new events buried in sensitive data, such as transaction records in banking or patient care journey in healthcare. No matter how good a model is, if the training data is not intelligent.  

Injecting new domain knowledge into models is also problematic. Due to regulations, customer data often cannot be linked to other, even publicly available data sources. Without the ability to add new knowledge into models, their intelligence will be limited. 

The majority of AI/ML projects never make it into production due to the lack of high-quality training data. Most organizations do have the data, but the intelligence is locked up. Data owners are unwilling to provide the necessary training data for security and compliance reasons. Even if data owners are on board, legacy anonymization used in data preparation destroys data utility. Old anonymization tools strip away the intelligence, ultimately leading to wrong business decisions.

To make matters worse, automated bias is creeping up on the data-driven world. According to Gartner, an estimated 85% of algorithms are by now erroneous due to bias. Currently, a lot of companies simply delete data with gender and race information. However, this won’t remove the bias, only make biased AI decisions harder to catch.

In AI/ML development, synthetic training data is better than real data. Not only is it perfectly privacy-compliant, but due to the nature of the AI-powered synthesization process, the original data can be augmented. For example, rare patterns and events can be upsampled in the synthetic training data. AI performance improves as much as 15% when training flexible capacity models on synthetic data. Synthesization can also create more records to fix embedded biases. For example, you can generate more high earning women than what is to be found in the original. The result is fair synthetic data, a must-have for responsible AI development. Injecting new domain knowledge into models becomes possible too, for example by adding synthetic geo data to risk prediction models. Synthetic data will also be the foundation on which Explainable AI will be built, providing insight into the decisions models make.