Submit a Distributed Training (MPI) Run:AI job for execution
runai submit-mpi job-name [--always-pull-image] [--args stringArray] [--backoffLimit int] [--command stringArray] [--cpu double] [--cpu-limit double] [--environment stringArray | -e stringArray] [--gpu int | -g int] [--host-ipc] [--host-network] [--image string | -i string] [--interactive] [--large-shm] [--local-image] [--memory string] [--memory-limit string] [--node-type string] [--processes int] [--pvc [StorageClassName]:Size:ContainerMountPath:[ro]] [--run-as-user] [--volume stringArray | -v stringArray] [--working-dir] . [--loglevel string] [--project string | -p string] [--help | -h]
- Options with value type of stringArray mean that you can add multiple values. You can either separate values with a comma or add the flag twice.
<job-name> the name of the job.
When starting a container, always pull the image from registry, even if cached on running node. This is useful when you are re-saving updates to the image using the same tag.
Arguments to pass to the command run on container start. Use together with
--command sleep --args 10000
The number of times the job will be retried before failing. The default is 6. This flag will only work with training workloads (when the
--interactiveflag is not specified)
Command to run at container start. Use together with
CPU units to allocate for the job (0.5, 1, .etc). The Job will receive at least this amount of CPU. Note that the Job will not be scheduled unless the system can guarantee this amount of CPUs to the job.
Limitations on the number of CPU consumed by the job (0.5, 1, .etc). The system guarantees that this Job will not be able to consume more than this amount of GPUs.
-e stringArray | --environment stringArray
Define environment variables to be set in the container. To set multiple values add the flag multiple times (
-e BATCH_SIZE=50 -e LEARNING_RATE=0.2) or separate by a comma (
--gpu int | -g int
Number of GPUs to allocate to the Job. The default is no GPUs.
Use the host's ipc namespace. Controls whether the pod containers can share the host IPC namespace. IPC (POSIX/SysV IPC) namespace provides separation of named shared memory segments, semaphores and message queues. Shared memory segments are used to accelerate inter-process communication at memory speed, rather than through pipes or through the network stack
For further information see docker documentation
Use the host's networkstack inside the container For further information see docker documentation
--image string | -i string
Image to use when creating the container for this Job
Mark this Job as Interactive. Interactive jobs are not terminated automatically by the system
Mount a large /dev/shm device. shm is a shared file system mounted on RAM
Use a local image for this job. A local image is an image which exists on all local servers of the Kubernetes Cluster.
CPU memory to allocate for this job (1G, 20M, .etc).The Job will receive at least this amount of memory. Note that the Job will not be scheduled unless the system can guarantee this amount of memory to the job.
CPU memory to allocate for this job (1G, 20M, .etc).The system guarantees that this Job will not be able to consume more than this amount of memory. The Job will receive an error when trying to allocate more memory than this limit.
Allows defining specific Nodes (machines) or group of Nodes on which the workload will run. To use this feature your administrator will need to label nodes as explained here: Limit a Workload to a Specific Node Group This flag can be used in conjunction with Project-based affinity. In this case, the flag is used to refine the list of allowable node groups set in the project. For more information see: Working with Projects
Number of distributed training processes. The default is 1.
Mount a persistent volume claim into a container
StorageClassName is a storage class name which can be obtained by running
kubectl get storageclasses.storage.k8s.io. This parameter may be omitted if there is a single storage class in the system, or you are using the default storage class.
Size is the volume size you want to allocate. See Kubernetes documentation for how to specify volume sizes
ContainerMountPath. A path internal to the container where the storage will be mounted
--pvc :3Gi:/tmp/john:ro- Allocate 3GB from the default Storage class. Mount it to /tmp/john as read-only
--pvc my-storage:3Gi:/tmp/john:ro- Allocate 3GB from the my-storage storage class. Mount it to /tmp/john as read-only
--pvc :3Gi:/tmp/john- Allocate 3GB from the default storage class. Mount it to /tmp/john as read-write
Run in the context of the current user running the Run:AI command rather than the root user. While the default container user is root (same as in Docker), this command allows you to submit a job running under your linux user. This would manifest itself in access to operating system resources, in the owner of new folders created under shared directories etc.
--volume stringArray | -v stringArray
Volume to mount into the container. Example
-v /raid/public/john/data:/root/data:roThe flag may optionally be suffixed with
:rwto mount the volumes in read-only or read-write mode, respectively.
Starts the container with the specified directory
Set the logging level. One of: debug|info|warn|error (default "info")
--project | -p (string)
Specify the project to which the command applies. Run:AI Projects are used by the scheduler to calculate resource eligibility. By default, commands apply to the default project. To change the default project use
runai project set <project-name>.
--help | -h
Show help text
start an unattended mpi training job of name dist1, based on project team-a using a quickstart-distributed image:
runai submit-mpi dist1 --num-processes=2 -g 1 \ -i gcr.io/run-ai-demo/quickstart-distributed
The command will attempt to submit an mpi job. You can follow up on the job by running
runai list or
runai get job-name
- See Walk-through document Walk-through: Running Distributed Training.