Use Apache Hive for synthetic data

MOSTLY AI can use Apache Hive as a data source for original data as well as a destination for synthetic data. To do so, you need to create Apache Hive connectors.

Create an Apache Hive connector

For each Apache Hive data source or destination, you need a separate connector.

Prerequisites

Obtain the Apache Hive connection details. To use Kerberos, see Use Kerberos for authentication.

• host
• port
• credentials

c = mostly.connect({
    "name": "Apache Hive Connector",
    "type": "HIVE",
    "accessType": "SOURCE",
    "config": {
        "host": "hive.hostname.com",
        "port": "10000",
        "username": "mostlyai",
    },
    "secrets": {
        "password": "******"
    },
    "testConnection": True
})

Use Kerberos for authentication

To create an Apache Hive connector with Kerberos authentication, contact your Kerberos system administrator to obtain the information listed below.

• Kerberos principal. A unique identity to which Kerberos can assign tickets.
• Kerberos krb5.conf configuration file. The contents of a krb5.conf configuration file, such as the one listed below. For more information, see krb5.conf (opens in a new tab) in the MIT Kerberos Documentation.

  [libdefaults]
  default_realm = INTERNAL
  dns_lookup_realm = false
  dns_lookup_kdc = false
  forwardable = true
  rdns = false
  
  [realms]
  INTERNAL = {
  	kdc = ip-172-***-***-222
  	kdc = ip-172-***-***-222.hive-kerberized.domain.com
  	kdc = 172.***.***.222
  	kdc = 3.***.***.111
  	kdc = hive-kerberized.domain.com
  	admin_server = hive-kerberized.domain.com
  }
  
  [domain_realm]
  .hive-kerberized.domain.com = INTERNAL
  hive-kerberized.domain.com = INTERNAL
• Kerberos keytab. Short for "key table", keytab files are used in Kerberos authentication to store keys needed to log in to Kerberos-aware services. Keytab files allow automated processes (scripts and service authentication) to authenticate using Kerberos without requiring a human to enter a password.
• CA certificate. If you have access to your keystore file from which you want to extract the CA certificate, you can use the following command to extract the CA certificate in PEM format.
  shell

  keytool -exportcert -alias ALIAS -keystore KEYSTORE_FILE -storepass STORE_PASSWORD -file ca-certificate.pem -rfc
  For example:
  shell

  keytool -exportcert -alias hivessl-ss -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/hivessl-ss.p12 -storepass 123456 -file /tmp/ca-cert.pem -rfc

# Import required Python libraries and initialize MostlyAI() client
import pandas as pd
import base64
from mostlyai import MostlyAI
mostly = MostlyAI(api_key=INSERT_API_KEY)
 
# Read the contents of your Kerberos keytab file and encode them to a base64 string
keytab_data = open('kerberos/hive.keytab', 'rb').read()
base64_keytab_string = base64.b64encode(keytab_data).decode('utf-8')
 
# Read the contents of your CA certificate PEM file. It should already be base64-encoded
base64_ssl_cert_string = open('kerberos/ca_cert.pem', 'rb').read()
 
# Create a multi-line string with the contents of your krb5.conf file
# Note: Contains examples of configuration values some of which are masked with asterisks (*)
kerberos_krb5_conf = """
[libdefaults]
  default_realm = INTERNAL
  dns_lookup_realm = false
  dns_lookup_kdc = false
  forwardable = true
  rdns = false
 
[realms]
	INTERNAL = {
		kdc = ip-172-***-***-222
		kdc = ip-172-***-***-222.hive-kerberized.domain.com
		kdc = 172.***.***.222
		kdc = 3.75.170.111
		kdc = hive-kerberized.domain.com
		admin_server = hive-kerberized.domain.com
	}
 
[domain_realm]
    .hive-kerberized.domain.com = INTERNAL
    hive-kerberized.domain.com = INTERNAL
"""
 
# Create a `config` variable with the contents of your Apache Hive connector configuration
config = {
    "name": "Apache Hive Connector with Kerberos auth",
    "type": "HIVE",
    "accessType": "SOURCE",
    "config": {
        "host": "hive-kerberized.test.mostlylab.com",
        "port": 10000,
        "kerberosEnabled": True,
        "kerberosPrincipal": "hive/hive-kerberized.test.mostlylab.com@INTERNAL",
        "kerberosKrb5Conf": kerberos_krb5_conf,
        "sslEnabled": True
    },
    "secrets": {
        "kerberosKeytab": base64_keytab_string
    },
    "ssl": {
        "caCertificate": base64_ssl_cert_string
    },
    "testConnection": True
}
 
# Create the connector
c = mostly.connect(config=config)

What's next

Depending on whether you created a source or a destination connector, you can use the connector as:

• data source for a new generator
• data destination for a new synthetic dataset

Considerations for Apache Hive as a destination

The Apache Hive connector uses a database interface to connect to your Apache Hive instance. While the database interface provides high-performance for read operations (for example, to pull original data for synthesis), using that interface to load synthetic data in batches can be too slow. It is not a recommended solution for delivering high volumes of synthetic data.

The fastest way to load data into Apache Hive is to save synthetic data as Parquet files into an S3 storage bucket and then load the data into Apache Hive.

For example:

CREATE TABLE table_name;
LOAD DATA INPATH 's3://bucket-name/path/to/parquet-file' INTO TABLE table_name;