Clusters

Description: When the cluster's status is ‘disconnected’, there is no communication from the cluster services reaching the Run:ai Platform. This may be due to networking issues or issues with Run:ai services.

Mitigation:

1. Check Run:ai’s services status:

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permission to view pods
   • Copy and paste the following command to verify that Run:ai’s services are running:

   kubectl get pods -n runai | grep -E 'runai-agent|cluster-sync|assets-sync'
   

   * If any of the services are not running, see the ‘cluster has service issues’ scenario.

2. Check the network connection

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permissions to create pods
   • Copy and paste the following command to create a connectivity check pod:

   kubectl run control-plane-connectivity-check -n runai --image=wbitt/network-multitool \
       --command -- /bin/sh -c 'curl -sSf <control-plane-endpoint> > /dev/null && echo "Connection Successful" \
       || echo "Failed connecting to the Control Plane"'
   

   • Replace <control-plane-endpoint> with the URL of the Control Plane in your environment. If the pod fails to connect to the Control Plane, check for potential network policies

3. Check and modify the network policies

   • Open your terminal

   • Copy and paste the following command to check the existence of network policies:
     

     kubectl get networkpolicies -n runai
     

   • Review the policies to ensure that they allow traffic from the Run:ai namespace to the Control Plane. If necessary, update the policies to allow the required traffic. Example of allowing traffic:

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
   name: allow-control-plane-traffic
   namespace: runai
   spec:
   podSelector:
       matchLabels:
       app: runai
   policyTypes:
       - Ingress
       - Egress
   egress:
       - to:
           - ipBlock:
               cidr: <control-plane-ip-range>
       ports:
           - protocol: TCP
           port: <control-plane-port>
   ingress:
       - from:
           - ipBlock:
               cidr: <control-plane-ip-range>
       ports:
           - protocol: TCP
           port: <control-plane-port>
   

   • Check infrastructure-level configurations:

     • Ensure that firewall rules and security groups allow traffic between your Kubernetes cluster and the Control Plane
     • Verify required ports and protocols:
       • Ensure that the necessary ports and protocols for Run:ai’s services are not blocked by any firewalls or security groups

4. Check Run:ai services logs

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permissions to view logs
   • Copy and paste the following commands to view the logs of the Run:ai services:

   kubectl logs deployment/runai-agent -n runai
   kubectl logs deployment/cluster-sync -n runai
   kubectl logs deployment/assets-sync -n runai
   

   • Try to identify the problem from the logs. If you cannot resolve the issue, continue to the next step.

5. Contact Run:ai’s support

   • If the issue persists, contact Run:ai’s support for assistance.

Description: When a cluster's status is Has service issues, it means that one or more Run:ai services running in the cluster are not available.

Mitigation:

1. Verify non-functioning services

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permissions to view the runaiconfig resource
   • Copy and paste the following command to determine which services are not functioning:

   kubectl get runaiconfig -n runai runai -ojson | jq -r '.status.conditions | map(select(.type == "Available"))'
   

2. Check for Kubernetes events

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permissions to view events
   • Copy and paste the following command to get all Kubernetes events:

3. Inspect resource details

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permissions to describe resources
   • Copy and paste the following command to check the details of the required resource:

   kubectl describe <resource_type> <name>
   

4. Contact Run:ai’s Support

   • If the issue persists, contact contact Run:ai’s support for assistance.

Description: When the cluster's status is ‘waiting to connect’, it means that no communication from the cluster services reaches the Run:ai Platform. This may be due to networking issues or issues with Run:ai services.

Mitigation:

1. Check Run:ai’s services status

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permissions to view pods
   • Copy and paste the following command to verify that Run:ai’s services are running:

   kubectl get pods -n runai | grep -E 'runai-agent|cluster-sync|assets-sync'
   

   • If any of the services are not running, see the ‘cluster has service issues’ scenario.

2. Check the network connection

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permissions to create pods
   • Copy and paste the following command to create a connectivity check pod:

   kubectl run control-plane-connectivity-check -n runai --image=wbitt/network-multitool --command -- /bin/sh -c 'curl -sSf <control-plane-endpoint> > /dev/null && echo "Connection Successful" || echo "Failed connecting to the Control Plane"'
   

   • Replace <control-plane-endpoint> with the URL of the Control Plane in your environment. If the pod fails to connect to the Control Plane, check for potential network policies:

3. Check and modify the network policies

   • Open your terminal
   • Copy and paste the following command to check the existence of network policies:

   kubectl get networkpolicies -n runai
   

   • Review the policies to ensure that they allow traffic from the Run:ai namespace to the Control Plane. If necessary, update the policies to allow the required traffic. Example of allowing traffic:

   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
   name: allow-control-plane-traffic
   namespace: runai
   spec:
     podSelector:
       matchLabels:
       app: runai
     policyTypes:
       - Ingress
       - Egress
     egress:
       - to:
           - ipBlock:
               cidr: <control-plane-ip-range>
       ports:
           - protocol: TCP
           port: <control-plane-port>
     ingress:
       - from:
           - ipBlock:
               cidr: <control-plane-ip-range>
       ports:
           - protocol: TCP
           port: <control-plane-port>
   

   • Check infrastructure-level configurations:
   • Ensure that firewall rules and security groups allow traffic between your Kubernetes cluster and the Control Plane
   • Verify required ports and protocols:
     • Ensure that the necessary ports and protocols for Run:ai’s services are not blocked by any firewalls or security groups

4. Check Run:ai services logs

   • Open your terminal
   • Make sure you have access to the Kubernetes cluster with permission to view logs
   • Copy and paste the following commands to view the logs of the Run:ai services:

   kubectl logs deployment/runai-agent -n runai
   kubectl logs deployment/cluster-sync -n runai
   kubectl logs deployment/assets-sync -n runai
   

   • Try to identify the problem from the logs. If you cannot resolve the issue, continue to the next step

5. Contact Run:ai’s support

   • If the issue persists, contact Run:ai’s support for assistance.

Description: When a cluster's status displays Missing prerequisites, it indicates that at least one of the Mandatory Prerequisites has not been fulfilled. In such cases, Run:ai services may not function properly.

Mitigation:

If you have ensured that all prerequisites are installed and the status still shows missing prerequisites, follow these steps:

1. Check the message in the Run:ai platform for further details regarding the missing prerequisites.

2. Inspect the runai-public ConfigMap:

   • Open your terminal. In the terminal, type the following command to list all ConfigMaps in the runai namespace:

   kubectl get configmap -n runai
   

3. Describe the ConfigMap

   • Locate the ConfigMap named runai-public from the list
   • To view the detailed contents of this ConfigMap, type the following command:

   kubectl describe configmap runai-public -n runai
   

4. Find Missing Prerequisites

   • In the output displayed, look for a section labeled dependencies.required
   • This section provides detailed information about any missing resources or prerequisites. Review this information to identify what is needed

5. Contact Run:ai’s support

   • If the issue persists, contact Run:ai’s support for assistance.