Researcher Library: Extended Reporting on Workload Progress¶

The Run:ai Researcher Library is a python library you can add to your deep learning python code. The reporting module in the library will externalize information about the run which can then be available for users of the Run:ai user interface.

With the reporter module, you can externalize information such as progress, accuracy, and loss over time/epoch, and more. In addition, you can externalize custom metrics of your choosing.

Sending Metrics¶

Python Deep-Learning Code¶

In your command-line run:

pip install runai

In your python code add:

import runai.reporter

Create a Reporter object as a Python context manager (i.e. with a with statement):

with runai.reporter.Reporter() as reporter:
    pass

Then use reporter to send metrics and parameters.

To send a number-based metric report, write:

reporter.reportMetric(<reporter_metric_name>, <reporter_metric_value>)

For example,

reporter.reportMetric("accuracy", 0.34)

To send a text-based metric report, write:

reporter.reportParameter(<reporter_param_name>, <reporter_param_value>)

For example,

reporter.reportParameter("state", "Training Model")

Recommended Metrics to send¶

For the sake of uniformity with the Keras implementation (see below), we recommend sending the following metrics:

Metric	Type	Frequency of Send	Description
accuracy	numeric	Each step	Current accuracy of run
loss	numeric	Each step	Current result of loss function of run
learning_rate	numeric	Once	Defined learning rate of run
step	numeric	Each Step	Current step of run
number_of_layers	numeric	Once	Number of layers defined for the run
optimizer_name	text	Once	Name of Deep Learning Optimizer
batch_size	numeric	Once	Size of batch
epoch	numeric	Each epoch	Current Epoch number
overall_epochs	numeric	Once	Total number of epochs

epoch and overall_epochs are especially important since the Job progress bar is computed by dividing these parameters.

Automatic Sending of Metrics for Keras-Based Scripts¶

For Keras based deep learning runs, there is support to automate the task of sending metrics.

First, import runai.reporter.keras instead of runai.reporter. Second, use runai.reporter.keras.Reporter instead of runai.reporter.Reporter.

The Keras reporter supports automatic logging. This could be done in two ways: 1. Passing autolog=True upon creation 2. Calling the method autolog after creation

After enabling automatic logging, the above metrics will automatically be sent going forward.

For example:

with runai.reporter.keras.Reporter(autolog=True) as reporter:
    pass

Adding the Metrics to the User interface¶

The metrics show up in the Job list of the user interface. To add a metric to the UI

Integrate the reporter library into your code
Send metrics via the reporter library
Run the workload once to send initial data.
Open the Run:ai user interface. Go to Jobs.
On the top right, use the settings wheel and select the metrics you have added

Trying it out with Quickstart¶

To try image reporting using the Run:ai Quickstart image use the RUNAI_REPORT flag.

Specifically, run:

runai submit -g 1 -i gcr.io/run-ai-demo/quickstart -e RUNAI_REPORT=1

Last update: March 23, 2022