Data sharing: fix broken data access with synthetic data

• Data access is increasingly limited within organizations. Data access privileges are getting hard to come by, and rightly so. According to Gartner: 

"59% of privacy incidents originate with an organization's own employees. Worse still — 45% of employee-driven privacy failures come from intentional behavior (though it may not be malicious)."

• Limiting attack surfaces has become a high priority for companies that suffer major financial and reputational setbacks when data leaks happen. Protecting perimeters is no longer enough. Reducing the amount of unsafe data within the walls of organizations is more important than ever. 
• However, most traditional data governance strategies are not only unsafe, but seriously inefficient, with data scientists spending 80% of their time finding, cleaning, and organizing data. 
• In addition, more and more external stakeholders could benefit from easier access to data. Vendors and start-ups are asking for your data to work with. Research partners want access as well. Off-shore development teams rely heavily on data sharing for testing applications.
• With an increase in data privacy legislations and rulings, like Schrems II effectively prohibiting US-EU data sharing, such projects turn out to be impossible to pull off. An increasingly hostile cybersecurity environment further inhibits free data flows making organizations more reluctant than ever to share data.

Everyone is talking about the importance of data-driven decisions, but in reality only a select few individuals in organizations actually have the data to make those decisions. Many times only privileged data scientists have full access to raw data. But it's not always easy for them either: often they need to request specific permissions to work with certain datasets.

Once data scientists or machine learning engineers venture into yet-undiscovered territories and ideas, they need to obtain new permissions. Sometimes that is even the case for performing new analyses on datasets they already worked with in the past! Depending on the organization these processes to gain permission can take weeks or more.

When it comes to external data sharing, organizations, especially those handling troves of sensitive data, like financial institutions, banks and insurance companies have two options: either to not share data externally at all, or to heavily rely on legacy data anonymization approaches. These approaches are known for their poor privacy protection and often poor data utility as well. Even worse, less mature organizations take unacceptable levels of risk by relying on simple forms of de-identification or sharing production data.

Better, faster and compliant ways of data access are already possible today with the right approach, yet most companies lack the awareness of: synthetic data.

Data is increasingly treated as a product, even and especially within the walls of organizations. Data should be proactively served in a cross-departmental fashion, flowing freely between different lines of business and even subsidiaries located in different countries or continents.

The much-coveted concept of the data mesh remains hard to attain for highly regulated industries without the necessary privacy-enhancing technologies. And there is one privacy-enhancing technology, that stands out: synthetic data. It is revolutionizing data anonymization and data-sharing processes and making true data democratization an everyday reality. 

In practical terms, the use of synthetic data significantly simplifies the implementation of data democratization within an organization, especially in sectors subject to stringent regulatory guidelines, such as healthcare, banking, and government.

While traditional data-sharing methods often require lengthy approval processes and complex legal frameworks to ensure privacy and compliance, synthetic data can bypass these hurdles. This is because synthetic data retains the useful characteristics of the original dataset for analysis, learning, or decision-making, but doesn't carry the personal or sensitive information that would trigger privacy concerns.

Therefore, synthetic data can be shared more freely across various departments, business units, or even between different companies in a conglomerate, without necessitating exhaustive privacy impact assessments or risking regulatory fines.

This not only speeds up decision-making but also fosters a more collaborative and innovative work environment. With synthetic data, the aspirational concept of a data mesh—a decentralized, domain-oriented ownership model for data architecture—becomes not just achievable but operationally efficient, even in the most regulated industries.

See a concrete example of how synthetic data can be shared within an Databricks environment in the following video.

More and more companies pivot to a proactive data approach. These innovators create internal - or in some cases, external data exchange platforms - to facilitate innovation and data-forward thinking across the organization and beyond.

Synthetic data sandboxes are populated with curated and maintained synthetic versions of business-critical datasets. Access to synthetic data assets can be broadly and quickly provided. Citizen data scientists can freely use synthetic data sandboxes, accelerating innovation and compliance. This helps to unlock customer data for a wide variety of further use cases, such as: 

• advanced analytics, AI and machine learning model training, 
• test data for software development.

McKinsey estimates that privacy-safe data sharing could generate almost $3 trillion annual economic value. And synthetic data generators are the technology to make this a reality.