InstallationDeployDeploy to minikube

Deploy MOSTLY AI to a minikube cluster

One of the options to install MOSTLY AI without the need of a multi-node Kubernetes cluster is to do so on a local Kubernetes cluster with minikube. More specifically, this page covers the steps to install MOSTLY AI on a Ubuntu Server LTS.

Prerequisites

  • A virtual machine or server that meets the minimum requirements.
    • Ubuntu Server LTS. For example, in terms of an AWS EC2 instance, your virtual machine could be of the type m5a.8xlarge
    • 24 vCPUs minimum
    • 48GB RAM minimum
    • 150-200GB disk spae
  • Microsoft Remote Desktop application

Pre-deploy

Task 1: Add the Docker repository to APT sources

  1. Install the packages needed to use a repository over HTTPS.
    sudo apt-get install ca-certificates curl gnupg lsb-release
  2. Add the official Docker GPG key.
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
  3. Add the Docker repository to APT sources.
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  4. Update the apt package index.
    sudo apt-get update

Task 2: Install Docker Engine

  1. Install Docker, containerd, and Docker Compose.
    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
  2. Start Docker.
    sudo systemctl start docker

Task 3: Install minikube

  1. Download the minikube Linux binaries.
    curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
  2. Install the minikube binaries in /usr/local/bin/minikube.
    sudo install minikube-linux-amd64 /usr/local/bin/minikube

Task 4: Install kubectl

  1. Download and install the kubectl Linux binaries.
    sudo snap install kubectl --classic
  2. Check the kubectl version.
    kubectl version --client
    The output should be similar to the following:
    Client Version: v1.28.3
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

Task 5: Install Ubuntu Desktop

  1. Switch from the current user to the super user account su.
    sudo su -
  2. Change the password for the ubuntu user.
    passwd ubuntu
  3. Update the APT package index.
    sudo apt update
  4. Install the Ubuntu Desktop package.
    💡

    The dependency list for the Ubuntu Desktop is long. It might take several minutes to install all dependent packages.

    sudo apt install ubuntu-desktop
  5. Install the xrdp package to enable graphical login via RDP (Microsoft Remote Desktop Protocol).
    sudo apt install xrdp
  6. Verify the installation of xrdp.
    sudo systemctl status xrdp
    The output from the command should be similar to the following:
    ● xrdp.service - xrdp daemon
        Loaded: loaded (/lib/systemd/system/xrdp.service; enabled; vendor preset: enabled)
        Active: active (running) since Tue 2023-11-07 20:05:16 UTC; 19s ago
        Docs: man:xrdp(8)
                man:xrdp.ini(5)
        Process: 32722 ExecStartPre=/bin/sh /usr/share/xrdp/socksetup (code=exited, status=0/SUCCESS)
        Process: 32730 ExecStart=/usr/sbin/xrdp $XRDP_OPTIONS (code=exited, status=0/SUCCESS)
    Main PID: 32731 (xrdp)
        Tasks: 1 (limit: 75773)
        Memory: 856.0K
            CPU: 16ms
        CGroup: /system.slice/xrdp.service
                └─32731 /usr/sbin/xrdp
  7. Allow connections through the Remote Desktop port 3389.
    sudo ufw allow 3389
    The output from the command should be similar to the following:
    Rules updated
    Rules updated (v6)

Task 6: Allow RDP connection to the machine

This task describes the process to allow RDP connections to an AWS EC2 instance.

Steps

  1. In AWS, open EC2 and open your instance details.
  2. Select the Security tab from the bottom.
  3. Click the Security group for the instance. AWS EC2 Security - Select security group
  4. For Inbound rules at the bottom, click Edit inbound rules. AWS EC2 Security - Click Edit inbound rules
  5. For Type, select RDP and for Source, select 0.0.0.0/0.
  6. Click Save rules. AWS EC2 Security - Select type and CIDR

Result

You can now use Remote Desktop to connect to your machine.

Task 7: Connect to the machine using Remote Desktop

This task describes the process to use Microsoft Remote Desktop for macOS. The same process should be similar on any other operating system.

Prerequisites

Obtain the public IPv4 address or DNS of your machine. For example, for an EC2 instance, you can copy the value from the Public IPv4 DNS or the Public IPv4 address fields.

AWS - Obtain instance IPv4 or DNS name

Steps

  1. Open the Remote Desktop application.
  2. Select Add PC.
  3. Paste the IP address or DNS of your machine and click Add.
  4. Double-click the new machine or right-click and select Connect. AWS - Click Connect for an EC2 instance
  5. Enter the ubuntu username and password your defined in Task 5.
  6. Enter the password again at the authentication prompt.

Result

You can now start a Terminal app and continue with Ubuntu Desktop.

Task 8: Start minikube

From the Ubuntu Terminal app, start minikub following the steps below.

Prerequisites

Add the docker user to the docker group.

sudo usermod -aG docker $USER && newgrp docker

Steps

  1. Start a minikube cluster.
    minikube start --driver=docker --memory 24000 --cpus 14
    💡

    Tip
    Use the provided values of 24 GB memory and 14 CPU cores as the minimum requirements to set the default compute required by MOSTLY AI. For all other purposes, increase the values as necessary.

    The output from the command should be similar to the following:
    😀  minikube v1.31.2 on Ubuntu 22.04
    ✨  Using the docker driver based on user configuration
    📌  Using Docker driver with root privileges
    👍  Starting control plane node minikube in cluster minikube
    🚜  Pulling base image ...
    💾  Downloading Kubernetes v1.27.4 preload ...
        > preloaded-images-k8s-v18-v1...:  393.21 MiB / 393.21 MiB  100.00% 39.28 M
        > gcr.io/k8s-minikube/kicbase...:  447.62 MiB / 447.62 MiB  100.00% 35.84 M
    🔥  Creating docker container (CPUs=2, Memory=24000MB) ...
    🐳  Preparing Kubernetes v1.27.4 on Docker 24.0.4 ...
        ▪ Generating certificates and keys ...
        ▪ Booting up control plane ...
        ▪ Configuring RBAC rules ...
    📎  Configuring bridge CNI (Container Networking Interface) ...
        ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    🔍  Verifying Kubernetes components...
    🌟  Enabled addons: storage-provisioner, default-storageclass
    🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
  2. (Optional) Make docker the default driver. The next time you start minikube, you do not need to specify a driver.
    minikube config set driver docker

Task 9: Create a namespace to install MOSTLY AI

  1. Verify the installation of kubectl by listing your pods and namespaces.
    kubectl get po -A
    The output should be similar to the following listing.
    NAMESPACE     NAME                                READY     STATUS    RESTARTS    AGE
    kube-system   coredns-787d4945fb-qw4r8            1/1       Running   0           1m19s
    kube-system   etcd-minikube                       1/1       Running   0           1m32s
    kube-system   kube-apiserver-minikube             1/1       Running   0           1m33s
    kube-system   kube-controller-manager-minikube    1/1       Running   0           1m33s
    kube-system   kube-proxy-bx9kb                    1/1       Running   0           1m19s
    kube-system   kube-scheduler-minikub              1/1       Running   0           1m34s
    kube-system   storage-provisioner                 1/1       Running   0           1m31s
  2. Create a namespace for MOSTLY AI.
    kubectl create ns mostly-ai
  3. Verify the newly created mostly-ai namespace.
    kubectl get ns
    The resulting listing should be similar to the following.
    NAME              STATUS   AGE
    default           Active   3m
    kube-node-lease   Active   3m
    kube-public       Active   3m
    kube-system       Active   3m
    mostly-ai         Active   37s
  4. Set the mostly-ai namespace as the default.
    kubectl config set-context --current --namespace=mostly-ai
    If successful, the result from the command is the following:
    Context "minikube" modified.

Task 10: Install required minikube addons

  1. Install the NGINX add-on.
    minikube addons enable ingress
  2. Install the volume snapshots add-on.
    minikube addons enable volumesnapshots
  3. Install the CSI hostpath driver add-on.
    minikube addons enable csi-hostpath-driver
  4. Verify the installed add-ons.
    minikube addons list
    The output from the command is a table that should contain the highlight lines below:
    |-----------------------------|----------|--------------|--------------------------------|
    |         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
    |-----------------------------|----------|--------------|--------------------------------|
    | csi-hostpath-driver         | minikube | enabled   | Kubernetes                     |
    | ingress                     | minikube | enabled   | Kubernetes                     |
    | volumesnapshots             | minikube | enabled   | Kubernetes                     |
    |-----------------------------|----------|--------------|--------------------------------|

Task 11: Install Helm

  1. Add the Helm GPG key.
    curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
  2. Install the APT HTTPS transport to use repositories over HTTPS.
    sudo apt-get install apt-transport-https --yes
  3. Add the Helm repository to APT sources.
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
  4. Update the apt package index.
    sudo apt-get update
  5. Install Helm.
    sudo apt-get install helm

Deploy

Task 12: Start minikube dashboard

With minikube running, you can open the minikube dashboard in a web browser and observer changes to the minikube cluster.

Steps

  1. In the machine, open a Terminal app.
  2. Start the minikube dashboard.
    minikube dashboard

Result

In the default web browser, the minikube dashboard opens.

Azure - Select Network policy - None

Task 13: Define a local DNS name

MOSTLY AI requires a fully-qualified domain (FQDN) name where the application accepts HTTPS connections from a web browser. In a minikube environment, you cannot use a FQDN. Because you this, you can set up a local DNS hostname in the /etc/hosts file.

By default, minikube exposes applications with an IP address 192.168.49.2. You can map that for your local DNS hostname in /etc/hosts.

Steps

  1. Edit /etc/hosts.
    sudo vi /etc/hosts
  2. Add the following line.
    /etc/hosts
    127.0.0.1 localhost
    192.168.49.2 mostly.local

Result

When you now open mostly.local in a web browser, it will direct traffic to the MOSTLY AI app after you deploy it in the next section.

Task 14: Upload Helm chart

From your local computer, use scp to upload the MOSTLY AI Helm chart to the home folder of the machine that runs minikube.

Change KEYPAIR_FILENAME.pem to the actual key pair filename and IP_ADDRESS to the IPv4 address or DNS name of your machine.

scp -i KEYPAIR_FILENAME.pem MOSTLY_AI_Helm_chart.zip ubuntu@IP_ADDRESS:~

Task 15: Edit values.yaml

  1. On the machine, unzip the Helm chart.
  2. Edit the values.yaml file as highlighted below.
    1. At the start, set the application domain name to mostly.local. Do the same as listed below for minio.
      💡

      minio is the shared storage service.

      values.yaml
      _customerInstallation:
        domainNames:
          mostly-ai: &fqdn mostly.local
          minio: &fqdnMinio minio-mostly.local
    2. (Optional) Apply one of the configurations below depending on whether you intend to use SSL-encrypted access to the MOSTLY AI application.

      ➡️ You use an SSL certificate. Replace your-tls-secret with the TLS secret name as defined in your cluster configuration.
      💡

      Your IT department or Kubernetes administrator creates the FQDN and its SSL certificate and adds it to the configuration of your cluster. When added, it comes with a TLS secret name that you can define in the values.yaml file. For details, see Configure your domain SSL certificate.

      values.yaml
      _customerInstallation:
      ...
        deploymentSettings:
          tlsSecretName: &tlsSecretName your-tls-secret
      ...
      ➡️ You do not use an SSL certificate. Replace the your-tls-secret with an empty string and, for global.tls, set enabled to false.
      values.yaml
      _customerInstallation:
      ...
        deploymentSettings:
          tlsSecretName: &tlsSecretName [] # your-tls-secret
      ...
      global:
        ...
        tls:
          enabled: false
      ...
    3. (Optional) If you host third-party container images in an internal repository, replace docker.io in registryFor3rdPartyComponents.
      values.yaml
      _customerInstallation:
      ...
        deploymentSettings:
        ...
          registryFor3rdPartyComponents: &registryFor3rdPartyComponents docker.io
      ...
    4. (Optional) If you need to host MOSTLY AI container images in an internal repository, replace harbor.env.mostlylab.com/mostlyai in mostlyRegistry.
      values.yaml
      _customerInstallation:
      ...
        deploymentSettings:
        ...
          mostlyRegistry: &mostlyRegistry harbor.env.mostlylab.com/mostlyai
      ...
    5. (Optional) If you intend to use the MOSTLY AI image repository at harbor.env.mostlylab.com/mostlyai, set its secret in mostlyRegistryDockerConfigJson.
      💡

      To obtain the secret, contact your MOSTLY AI Customer Experience Engineer.

      values.yaml
      _customerInstallation:
      ...
        deploymentSettings:
        ...
          mostlyRegistryDockerConfigJson: &mostlyRegistryDockerConfigJson <HARBOR_SECRET>
      ...
    6. Define the Default compute under mostly-app.

      Use CPU and memory resources that are slightly below the ones defined for your minikube cluster. For example, if your minikube has 14 CPU cores and 24GB memory, define 10 CPU cores and 20GB memory in your Helm chart.
      values.yaml
      mostly-app:
      ...
          mostly:
            defaultComputePool:
              name: Default
              type: KUBERNETES
              toleration: engine-jobs
              resources:
                cpu: 10
                memory: 20
                gpu: 0
      ...
    7. Enable the ingress annotations. a. Enable the ingress annotations under mostly-app.
      values.yaml
      mostly-app:
          ingress:
              annotations:
                nginx.ingress.kubernetes.io/proxy-body-size: 10240m
                nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
                nginx.org/proxy-connect-timeout: 3000s
                nginx.org/proxy-read-timeout: 3000s
                nginx.org/client-max-body-size: 3000m
              # annotations: {}
      b. Enable the ingress annotations under mostly-keycloak.
      values.yaml
      mostly-keycloak:
          deployment:
              resources: {}
              tolerations: []
              affinity: {}
          ingress:
              annotations:
                  nginx.ingress.kubernetes.io/proxy-body-size: 10240m
                  nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
              # annotations: {}
  3. Save the values.yaml file.

Task 16: Add a taint to your compute nodes

The MOSTLY AI application requires that new compute jobs (generator training and synthetic data generations) run on Kubernetes compute nodes that have a specific taint.

Prerequisites

Contact your IT department or system administrator to check who can set taints on the Kubernetes nodes.

Steps

  • Set the taint on the only minikube node on which MOSTLY AI compute jobs will run.
    kubectl taint nodes minikube scheduling.mostly.ai/node=engine-jobs:NoSchedule

Result

You should see the following:

node/minikube tainted

Task 17: Deploy MOSTLY AI

Steps

  1. Make sure you are in the root folder of the Helm chart, where mostly-combined is located and is a subfolder.
  2. Use helm to deploy MOSTLY AI with the edited values.yaml file.
    helm upgrade --install mostly-ai ./mostly-combined --values values.yaml --namespace mostly-ai --create-namespace
  3. Open the web browser running the minikube dashboard.
  4. In the dashboard, select the mostly-ai namespace and then Workloads > Pods.

Result

Immediately after running the Helm command, you will see all pods (in the dashboard, go to Workloads > Pods) with different status.

Azure - Select Network policy - None

Wait for a while for all pods status to turn green.

Post-deployment

With the MOSTLY AI pods successfully started, you can now log in to your MOSTLY AI deployment for the first time.

Task 18: Log in to your MOSTLY AI deployment

Log in for the first time to your MOSTLY AI deployment to set a new password for the superadmin user.

Prerequisites

Contact MOSTLY AI to obtain the superadmin credentials. You need them to log in for the first time.

Steps

  1. Open the locally defined hostname mostly.local in a web browser.
    Step result: Your Sign in page for your MOSTLY AI deployment opens. MOSTLY AI Deployment - Log in page
  2. Enter the superadmin credentials and click Sign in.
  3. Provide a new password and click Change password.

Result

Your superadmin password is now changed and you can use it to log in again to your MOSTLY AI deployment.

What’s next

You can run a quick test by following the Quick start for Model creators and Quick start for Data consumers.