ConnectorsCreate a connectorS3 storage

Use S3 storage for synthetic data

If you keep datasets in S3 storage (Amazon S3 or any S3-compatible storage service), you can synthesize them via an S3 connector in MOSTLY AI.

If you want to store the generated synthetic data in a S3 separate bucket, you need to create a second destination S3 connector that points to that bucket.

Prerequisites

When you use AWS S3, take into account the prerequisites listed below.

  • Use only “long-term” credentials that include an access key and a secret key. “Short-term” credentials also require a session token, which is not supported.
  • To use AWS S3 paths containing partitioned Parquet datasets, your AWS credentials must have the s3:ListBucket permission.

Steps

  1. From the Connectors tab, click Create connector. Click Create connector button
  2. From the Create a new connector window, select S3 Storage. Select S3 Storage connector
  3. From the New connector window, configure the connector.
    1. For Name, enter a name that you can distinguish from other connectors.
    2. For Access type, select whether you want to use the connector as a source or destination.
    3. For Access key, enter your AWS access key.
    4. For Secret key, enter your AWS secret key. Configure AWS S3 bucket connector
    5. For Endpoint URL (optional), enter the endpoint URL of your S3-compatible storage service.
      💡

      If you use Amazon S3, you can leave this field empty.

      If you use a different S3-compatible storage service, enter the endpoint URL of the service. For example: https://play.min.io:9000.

      Configure AWS S3 bucket connector - Use SSL for encrypted connection
    6. (Optional) To use an encrypted connection, select Use SSL and upload your certificate in the CA certificate field.
  4. Click Save to save your new AWS storage connector.

    MOSTLY AI tests the connection. If you see an error, check the connection details, update them, and click Save again.

    You can click Save anyway to save the connector disregarding any errors.

What’s next

Depending on whether you created a source or a destination connector, you can use the connector as: