System Monitoring

This article explains how to configure Run:ai to generate health alerts and to connect these alerts to alert-management systems within your organization. Alerts are generated for Run:ai clusters.

Alert infrastructure¶

Run:ai uses Prometheus for externalizing metrics and providing visibility to end-users. The Run:ai Cluster installation includes Prometheus or can connect to an existing Prometheus instance used in your organization. The alerts are based on the Prometheus AlertManager. Once installed, it is enabled by default.

This document explains how to:

• Configure alert destinations - triggered alerts send data to specified destinations
• Understand the out-of-the-box cluster alerts, provided by Run:ai
• Add additional custom alerts

Prerequisites¶

• A Kubernetes cluster with the necessary permissions
• Up and running Run:ai environment, including Prometheus Operator
• kubectl command-line tool installed and configured to interact with the cluster

Set-up¶

Use the steps below to set up monitoring alerts.

Validating Prometheus operator installed¶

1. Verify that the Prometheus Operator Deployment is running
   Copy the following command and paste it in your terminal, where you have access to the Kubernetes cluster:
   kubectl get deployment kube-prometheus-stack-operator -n monitoring
   In your terminal, you can see an output indicating the deployment's status, including the number of replicas and their current state.
2. Verify that Prometheus instances are running
   Copy the following command and paste it in your terminal:
   kubectl get prometheus -n runai
   You can see the Prometheus instance(s) listed along with their status.

Enabling Prometheus AlertManager¶

In each of the steps in this section, copy the content of the code snippet to a new YAML file (e.g., step1.yaml).

• Copy the following command to your terminal, to apply the YAML file to the cluster:

kubectl apply -f step1.yaml
Copy the following command to your terminal to create the AlertManager CustomResource, to enable AlertManager:

apiVersion: monitoring.coreos.com/v1  
kind: Alertmanager  
metadata:  
   name: runai  
   namespace: runai  
spec:  
   replicas: 1  
   alertmanagerConfigSelector:  
      matchLabels:
         alertmanagerConfig: runai 

• Copy the following command to your terminal to validate that the AlertManager instance has started:
  kubectl get alertmanager -n runai
• Copy the following command to your terminal to validate that the Prometheus operator has created a Service for AlertManager:
  kubectl get svc alertmanager-operated -n runai

Configuring Prometheus to send alerts¶

1. Open the terminal on your local machine or another machine that has access to your Kubernetes cluster

2. Copy and paste the following command in your terminal to edit the Prometheus configuration for the runai Namespace:

   kubectl edit prometheus runai -n runai
   

   This command opens the Prometheus configuration file in your default text editor (usually vi or nano).

3. Copy and paste the following text to your terminal to change the configuration file:

   alerting:  
      alertmanagers:  
         - namespace: runai  
           name: alertmanager-operated  
           port: web
   

4. Save the changes and exit the text editor.

Alert destinations¶

Set out below are the various alert destinations.

Configuring AlertManager for custom email alerts¶

In each step, copy the contents of the code snippets to a new file and apply it to the cluster using kubectl apply -f.

Add your smtp password as a secret:

apiVersion: v1  
kind: Secret  
metadata:  
   name: alertmanager-smtp-password  
   namespace: runai  
stringData:
   password: "your_smtp_password"

Replace the relevant smtp details with your own, then apply the alertmanagerconfig using kubectl apply.

 apiVersion: monitoring.coreos.com/v1alpha1  
 kind: AlertmanagerConfig  
 metadata:  
   name: runai  
   namespace: runai  
 labels:  
    alertmanagerConfig: runai  
 spec:  
    route:  
       continue: true  
       groupBy:   
       - alertname

       groupWait: 30s  
       groupInterval: 5m  
       repeatInterval: 1h

    matchers:  
    - matchType: =~  
      name: alertname  
      value: Runai.*

    receiver: email

 receivers:  
 - name: 'email'  
   emailConfigs:  
   - to: '<destination_email_address>'  
     from: '<from_email_address>'  
     smarthost: 'smtp.gmail.com:587'  
     authUsername: '<smtp_server_user_name>'  
     authPassword:  
       name: alertmanager-smtp-password
         key: password  

Save and exit the editor. The configuration is automatically reloaded.

Third-party alert destinations¶

Prometheus AlertManager provides a structured way to connect to alert-management systems. There are built-in plugins for popular systems such as PagerDuty and OpsGenie, including a generic Webhook.

Example: Integrating Run:ai with a Webhook¶

1. Use webhook.site to get a unique URL.
2. Use the upgrade cluster instructions to modify the values file:
   Edit the values file to add the following, and replace <WEB-HOOK-URL> with the URL from webhook.site.

codekube-prometheus-stack:  
  ...  
  alertmanager:  
    enabled: true  
    config:  
      global:  
        resolve_timeout: 5m  
      receivers:  
      - name: "null"  
      - name: webhook-notifications  
        webhook_configs:  
          - url: <WEB-HOOK-URL>  
            send_resolved: true  
      route:  
        group_by:  
        - alertname  
        group_interval: 5m  
        group_wait: 30s  
        receiver: 'null'  
        repeat_interval: 10m  
        routes:  
        - receiver: webhook-notifications

Built-in alerts¶

A Run:ai cluster comes with several built-in alerts. Each alert notifies on a specific functionality of a Run:ai’s entity. There is also a single, inclusive alert: Run:ai Critical Problems, which aggregates all component-based alerts into a single cluster health test.

Runai agent cluster info push rate low

Runai cluster sync handling rate low

| Meaning | The cluster-sync Pod in the runai namespace might be functioning slowly | | | :---- |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Impact | Possible impact - info from the cluster is being synced back to the control-plane with a slow rate | | Severity | Warning | | Diagnosis | kubectl logs deployment/cluster-sync -n runai to see if the cluster-sync pod is running properly | | Troubleshooting/Mitigation | To diagnose issues with the cluster-sync pod, follow these steps: Check the Logs: Use the following command to view the logs of the cluster-sync deployment:kubectl logs deployment/cluster-sync -n runai Analyze the Logs and Pod Details: From the information provided by the logs and the deployment details, attempt to identify the reason why the cluster-sync pod is not functioning correctly Check Connectivity: Ensure there is a stable network connection between the cluster and the Run:ai Control Plane. A connectivity issue may be the root cause of the problem. Contact Support: If the network connection is stable and you are still unable to resolve the issue, contact Run:ai support for further assistance |

Runai agent pull rate low

Runai container memory usage critical

Runai container memory usage warning

Runai container restarting

Runai CPU usage warning

Runai critical problem

Runai daemonSet rollout stuck / Runai DaemonSet unavailable on nodes

Runai deployment insufficient replicas / Runai deployment no available replicas /RunaiDeploymentUnavailableReplicas

Runai project controller reconcile failure

Runai StatefulSet insufficient replicas / Runai StatefulSet no available replicas

Adding a custom alert¶

You can add additional alerts from Run:ai. Alerts are triggered by using the Prometheus query language with any Run:ai metric.

To create an alert, follow these steps using Prometheus query language with Run:ai Metrics:

• Modify Values File: Use the upgrade cluster instructions to modify the values file.
• Add Alert Structure: Incorporate alerts according to the structure outlined below. Replace placeholders <ALERT-NAME>, <ALERT-SUMMARY-TEXT>, <PROMQL-EXPRESSION>, <optional: duration s/m/h>, and <critical/warning> with appropriate values for your alert, as described below.

kube-prometheus-stack:  
   additionalPrometheusRulesMap:  
     custom-runai:  
       groups:  
       - name: custom-runai-rules  
         rules:  
         - alert: <ALERT-NAME>  
           annotations:  
             summary: <ALERT-SUMMARY-TEXT>  
           expr:  <PROMQL-EXPRESSION>  
           for: <optional: duration s/m/h>  
           labels:  
             severity: <critical/warning>

You can find an example in the Prometheus documentation.