Use a connector as a data source

If you use the web application, select a source connector to add data to a new generator.

Steps

1. From the Generators page, open an untrained generator or create a new one by clicking + New generator.
2. Click Add data.
3. Click Connect to source.
4. Select a source connector from the list from which to add data.
5. Follow the steps depending on whether you use a database connector or a cloud storage connector.
   • For cloud storage connectors, click in the Select table field to view and expand the bucket tree structure. Select a table or a folder containing a partitioned table (multiple .parquet files).
   • For database connectors, select a schema (optional) and a table from the list.
6. (Optional) Select Include child tables to add any tables that reference the currently selected table with a foreign key.

Result

The selected tables (and, if selected, any referencing child tables) are added to the generator.

If you use the Python client, add the name, location, and source_connector_id to the table dictionary in the tables list.

• name key - define the table name as it appears in the generator and in the synthetic datasets that you create with the generator
• location key
  • for databases with a schema, define the location as SCHEMA.TABLE
  • for databases without a schema, define the location as TABLE
  • for cloud buckets, define the location as the path to the table file/folder/subfolder/file.csv
• source_connector_id - ID of the source connector

Here is an Python example for a source database with a schema. Replace DB_WITH_SCHEMA_CONNECTOR_ID with the connector ID, SCHEMA with the schema name, and TABLE with the table name in the database.

config = {
    'name': 'US census',
    'tables': [
        {
            'name': 'census',
            'source_connector_id': 'DB_WITH_SCHEMA_CONNECTOR_ID',
            "location": "SCHEMA.TABLE",
        }
    ]
}
g = mostly.train(config=config)

For a source database with no schema. Replace DB_NO_SCHEMA_CONNECTOR_ID with the connector ID and TABLE with the table name in the database.

config = {
    'name': 'US census',
    'tables': [
        {
            'name': 'census',
            'source_connector_id': 'DB_NO_SCHEMA_CONNECTOR_ID',
            "location": "TABLE",
        }
    ]
}
g = mostly.train(config=config)

For a cloud bucket with a path to the table file.

config = {
    'name': 'US census',
    'tables': [
        {
            'name': 'census',
            'source_connector_id': 'CLOUD_BUCKET_CONNECTOR_ID',
            "location": "/FOLDER/FILE.CSV",
        }
    ]
}
g = mostly.train(config=config)