Researcher Library: Extended Reporting on Workload Progress¶
The Run:AI Researcher Library is a python library you can add to your deep learning python code. The reporting module in the library will externalize information about the run which can then be available for users of the Run:AI user interface (https://app.run.ai)
With the reporter module, you can externalize information such as progress, accuracy, and loss over time/epoch, and more. In addition, you can externalize custom metrics of your choosing.
Python Deep-Learning Code¶
In your command-line run:
pip install runai
In your python code add:
Reporter object as a Python context manager (i.e. with a
with runai.reporter.Reporter() as reporter: pass
reporter to send metrics and parameters.
To send a number-based metric report, write:
To send a text-based metric report, write:
reporter.reportParameter("state", "Training Model")
Recommended Metrics to send¶
For the sake of uniformity with the Keras implementation (see below), we recommend sending the following metrics:
|Metric||Type||Frequency of Send||Description|
|accuracy||numeric||Each step||accuracy of run|
|loss||numeric||Each step||result of loss function of run|
|learning_rate||numeric||Once||Defined learning rate of run|
|step||numeric||Each Step||Current step of run|
|number_of_layers||numeric||Once||Number of layers defined for the run|
|optimizer_name||text||Once||Name of Deep Learning Optimizer|
|batch_size||numeric||Once||Size of batch|
|epoch||numeric||Each epoch||Current Epoch number|
|overall_epochs||numeric||Once||Total number of epochs|
epoch and overall_epochs are especially important since the Job progress bar is computed by dividing these parameters.
Automatic Sending of Metrics for Keras-Based Scripts¶
For Keras based deep learning runs, there is support to automate the task of sending metrics.
runai.reporter.keras instead of
runai.reporter. Second, use
runai.reporter.keras.Reporter instead of
The Keras reporter supports automatic logging. This could be done in two ways: 1. Passing
autolog=True upon creation 2. Calling the method
autolog after creation
After enabling automatic logging, the above metrics will automatically be sent going forward.
with runai.reporter.keras.Reporter(autolog=True) as reporter: pass
Adding the Metrics to the User interface¶
The metrics show up in the Job list of the user interface. To add a metric to the UI
- Integrate the reporter library into your code
- Send metrics via the reporter library
- Run the workload once to send initial data.
- Go to Jobs list: https://app.run.ai/jobs
- On the top right, use the settings wheel and select the metrics you have added
Trying it out with Quickstart¶
To try image reporting using the Run:AI Quickstart image use the
runai submit -g 1 -i gcr.io/run-ai-demo/quickstart -e RUNAI_REPORT=1