Use the Numeric encoding type to synthesize numerical values that may vary, such as weight and height.

The Precision field appears when you select this type. Here, you can optionally limit the maximum number of digits after the decimal point used during training. The values will be rounded to the number of digits that you specified.

A categorical variable has a fixed set of possible values that are already present in the input data. An example of such a variable is T-shirt size, which consists of the following categories: 'XS, S, M, L, XL'. Categorical variables prevent random values (for instance, 'XM, A, B') from appearing in your synthetic dataset.

The Rare category protection settings appear below the Encoding type section when selecting this type. These settings help with protecting rare categories. Such categories may cause re-identification of outliers among your data subjects if they’re present in the resulting synthetic data.

There are two rare category protection methods available with which you can mask these categories:

With the Threshold parameter, you can specify when a category is considered rare.
A Threshold of 20 means that if a category is only present at 20 or fewer subjects, this category will be masked using the method you specified

The two charts below demonstrate how this works using the Baseball dataset. This dataset contains the records of 19,000 professional baseball players from 57 different countries, describing their country of origin, name, weight, height, etc.

Baseball is a popular sport in the U.S.A and some other countries in that region of the world. It’s therefore rare to find professional baseball players in European or Asian countries. The below chart — depicting the original dataset — reflects this distribution of baseball players. Over 16,000 baseball players come from the U.S.A, whereas there are only a few players in Belgium, Austria, or the Philippines.

You can prevent these baseball players from being identified by their country of origin by masking these categories. The below chart — depicting the synthetic dataset — shows the result of replacing them with the * label. Of the 58 countries in the original dataset are only 16 visible in the synthetic dataset. The subjects of the remaining 43 countries are in the new * category, preventing the re-identification of that sole baseball player in Greece, Indonesia, or Singapore.

Another use case for the categorical encoding type is postal codes (or ZIP codes). Specifying them as categorical rather than as a numeric column makes a big difference in the synthetic data generation process.

Most countries use numeric postal code systems. Only a few in the world are alphanumeric. MOSTLY AI automatically detects these as categories. With numeric systems, it’s likely to assign the Numeric encoding type.

We highly recommend verifying the encoding type for your postal code column. If the synthesization process uses the Numeric encoding type, unique postal codes may appear that aren’t present in the original dataset.

Datetime refers to values that contain a date part and a time part. This encoding type enables MOSTLY AI to synthesize them and generate valid and statistically representative dates and times.

The following formats are supported:

Use the Text encoding type to synthesize unstructured natural language texts up to 1000 characters long and for a maximum of 1 column per table.

You can use this encoding type to generate realistic, representative, and anonymous financial transaction texts, short user feedback, medical assessments, PII fields, etc. As the resulting synthetic texts are representative of the terms, tokens, and their co-occurrence in the original data, they can be confidently used in analytics and machine learning use cases, such as sentiment analysis and named-entity recognition. Even though they might look noisy and not very human-readable, they will work perfectly for these use cases.

Our text synthetization model is language-agnostic and doesn’t contain the biases of some pre-trained models—any content is solely learned from the original training data. This means that it can process any language, vernacular, and slang present in the original data.

The amount of data required to produce usable results depends on the diversity of the original texts' vocabulary, categories, names, etc. As a rule of thumb, the more structure there is, the fewer samples are needed.

The synthetic texts are generated in a context-aware manner—the messages from a teenager are different from those of an 85-year old grandmother, for instance. By considering the other attributes of a synthetic subject’s profile, MOSTLY AI is capable of synthesizing appropriate natural language texts for each of them.

Below, you can find two examples. The first example demonstrates MOSTLY AI’s ability to synthesize entirely new names from a multilingual dataset. And the second example shows the result of synthesizing Tripadvisor reviews. Here you can see that the resulting texts accurately retain the context of the establishment they discuss (Restaurant or Hotel) and the synthesized rating.

Multilingual names dataset

Tripadvisor reviews

ITT, or Inter-Transaction Time, is an encoding type that models the time interval between two subsequent events in the synthetic dataset. This encoding type causes the time between events to become very accurate, but the dates become less accurate.

Use the Latitude, Longitude encoding type to synthesize geolocation coordinates.

MOSTLY AI requires a geolocation coordinate to be encoded in a single field with the latitude and longitude as comma-separated values. The values must be in decimal degrees format and range from -90 to 90 for latitude and -180 to 180 for longitude. Their precision cannot be larger than five digits after the decimal dot. Any additional digits will be ignored.

The table below shows a use case with three geolocation columns.