lightbulb

Use this guide to diagnose and troubleshoot runtime issues that may occur when running MOSTLY AI.

list

MOSTLY AI needs to be installed before using this guide.

clock

It will take 15 mins to complete this guide.
You can find a description of MOSTLY AI’s architecture and its components, the locations of all configuration files, and the command-line instructions to check health statuses and retrieve logs.

MOSTLY AI’s components

Below you can see MOSTLY AI’s architecture diagram. It depicts how its different components interact with each other, with services in your company’s server environment, and with clients.


MOSTLY AI’s architecture


Application Nodes

Pod and image name Description Pod Lifecycle

browser code alt Web Frontend

mostly-app
mostlyai/mostly-app-enterprise

Contains MOSTLY AI’s frontend, backend and the API. Generally reachable over port 8080.

Always Up

database lock Coord. Service

mostly-coordinator
mostlyai/mostly-coordinator

This component takes all requests from the web application and coordinates execution of tasks on the main AI engine.

Always Up

portrait Keycloak

mostly-keycloak
mostlyai/mostly-keycloak

Keycloak is an open-source identity management, authentication, and authorization tool. This container has a pre-configured Keycloak instance for MOSTLY AI.

Always Up

database lock Rabbit MQ

mostly-rabbitmq
rabbitmq:3.8.14

Message queue handling communications between the AI engine and the application.

Always Up

database lock PostgreSQL

mostly-db
postgres:11

Database instance of the system. Has databases for app, coordinator and Keycloak.

Always Up

AI Worker nodes

Pod and image name Description Pod Lifecycle

pull request Task Agent job

mostly-agent
mostlyai/mostly-agent-cpu

MOSTLY AI’s AI engine.

Always Up

chip AI job

mostly-engine-v2
mostlyai/mostly-engine

The main engine component, which does the AI training and data generation.

On Demand

chip QA job

mostly-qa
mostlyai/mostly-analyze

Engine component, which creates the Quality Assurance report for privacy and accuracy of the generated data versus the original data.

On Demand

chip Data job

mostly-db-connector
mostlyai/mostly-db-connector

Engine component responsible for the database operations.

On Demand

Using the command-line utilities

MOSTLY AI comes with a couple of command line utilites called mostly-ai and mostly-agent, that are useful for conducting basic operations on the system.

They are already in the default path, but can be located in their respective folders:
/home/<user>/<installation_folder>/mostly-ai/bin
/home/<user>/<installation_folder>/mostly-agent/bin

You can use it to start, stop, or check MOSTLY AI’s status:

Start

mostly-ai start
Stop

mostly-ai stop
Check status

mostly-ai status

and

Start

mostly-agent start
Stop

mostly-agent stop
Check status

mostly-agent status

For the full list of commands, execute mostly-ai --help or mostly-agent --help respectively.

Configuration file locations

The following locations contain MOSTLY AI’s configuration files and working directory. You can find them in the installation folder, which is either /home/<user>/<installation_folder> or /opt/<installation_folder>.

Folder Location Description

etc

<installation_folder>/mostly-ai/etc

Configuration files

data

<installation_folder>/mostly-ai/data

Working directory

You can find the following configuration files in the etc folder:

Folder Description

docker-compose.yml

Contains container definitions, their relationships and integrations.

.env (hidden file)

Contains docker-compose.yml‘s environment variables.

Advanced troubleshooting

The commands below can help diagnose issues. Prefix them with sudo if the required privileges to run them are missing.

Container health status

Command

docker ps -all

Notes

Lists all the containers, when they were created and their status. If there is a missing container (see list of containers above), this indicates that it has stopped.

Also, if a container is constantly restarting (status always resetting to “Up x seconds”), this is an indication of a failure.

In such cases, stopping the application and restarting it using mostly-ai commands will help. If the issue persists, checking the logs is recommended.

Container logs

Command

docker logs <container_name>
e.g., docker logs mostly-agent

docker logs <container_id> e.g., docker logs 5c29f59982b9

Notes

Displays the logs of a container to the console.