Quickstart: Launch an Inference Workload¶

Introduction¶

Machine learning (ML) inference refers to the process of using a trained machine learning model to make predictions or generate outputs based on new, unseen data. After a model has been trained on a dataset, inference involves applying this model to new examples to produce results such as classifications, predictions, or other types of insights.

The quickstart below shows an inference server running the model and an inference client.

There are various ways to submit a Workload:

• Run:ai command-line interface (CLI)
• Run:ai user interface
• Run:ai API

At this time, Inference services cannot be created via the CLI. The CLI can be used for creating a client to query the inference service.

Prerequisites¶

To complete this Quickstart, the Infrastructure Administrator will need to install some optional inference prerequisites as described here.

To complete this Quickstart, the Platform Administrator will need to provide you with:

• ML Engineer access to Project in Run:ai named "team-a"
• The project should be assigned a quota of at least 1 GPU.
• The URL of the Run:ai Console. E.g. https://acme.run.ai.

As described, the inference client can be created via CLI. To perform this, you will need to have the Run:ai CLI installed on your machine. There are two available CLI variants:

• The older V1 CLI. See installation here
• A newer V2 CLI, supported with clusters of version 2.18 and up. See installation here

Step by Step Walkthrough¶

Login¶

Create an Inference Server Environment¶

To complete this Quickstart via the UI, you will need to create a new Inference Server Environment asset.

This is a one-time step for all Inference workloads using the same image.

Under Environments Select NEW ENVIRONMENT. Then select:

• A default (cluster) scope.
• Use the environment name inference-server.
• The image runai.jfrog.io/demo/example-triton-server.
• Under type of workload select inference.
• Under endpoint set the container port as 8000 which is the port that the triton server is using.

Run an Inference Workload¶

curl -L 'https://<COMPANY-URL>/api/v1/workloads/inferences' \ # (1)
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <TOKEN>' \ # (2)
-d '{ 
    "name": "inference-server-1", 
    "projectId": "<PROJECT-ID>", '\ # (3)
    "clusterId": "<CLUSTER-UUID>", \ # (4)
    "spec": {
        "image": "runai.jfrog.io/demo/example-triton-server",
        "servingPort": {
            "protocol": "http",
            "container": 8000
        },
        "autoscaling": {
            "minReplicas": 1,
            "maxReplicas": 2,
            "metric": "concurrency",
            "metricThreshold": 3,
            "scaleToZeroRetentionSeconds": 300
        },
        "compute": {
            "cpuCoreRequest": 0.1,
            "gpuRequestType": "portion",
            "cpuMemoryRequest": "100M",
            "gpuDevicesRequest": 1,
            "gpuPortionRequest": 0.5
        }
    }
}'

This would start a triton inference server with a maximum of 2 instances, each instance consumes half a GPU.

Query the Inference Server¶

You can use the Run:ai Triton demo client to send requests to the server

Find the Inference Server Endpoint¶

• Under Workloads, select Columns on the top right. Add the column Connections.
• See the connections of the inference-server-1 workload:

• Copy the inference endpoint URL.

runai project set team-a
runai training submit inference-client-1 -i runai.jfrog.io/demo/example-triton-client \
-- perf_analyzer -m inception_graphdef -p 3600000 -u <INFERENCE-ENDPOINT>    

runai config project team-a   
runai submit inference-client-1  -i runai.jfrog.io/demo/example-triton-client \
-- perf_analyzer -m inception_graphdef  -p 3600000 -u  <INFERENCE-ENDPOINT>    

In the user interface, under inference-server-1, go to the Metrics tab and watch as the various GPU and inference metrics graphs rise.

Stop Workload¶

Run the following: