Node Pools

This article explains the procedure for managing Node pools.

Node pools assist in managing heterogeneous resources effectively. A node pool is a Run:ai construct representing a set of nodes grouped into a bucket of resources using a predefined node label (e.g. NVidia GPU type) or an administrator-defined node label (any key/value pair).

Typically, the grouped nodes share a common feature or property, such as GPU type or other HW capability (such as Infiniband connectivity), or represent a proximity group (i.e. nodes interconnected via a local ultra-fast switch). Researchers and ML Engineers would typically use node pools to run specific workloads on specific resource types.

Platform administrators can create, view, edit, and delete node pools. Creating a new node pool creates a new instance of the Run:ai scheduler, workloads submitted to a node pool will be scheduled using the node pool’s designated scheduler instance.

Once a new node pool is created, it is automatically assigned to all Projects and Departments with a quota of zero GPU resources, unlimited CPU resources, and over-quota enabled (Medium priority if over-quota priority is enabled). This allows any Project and Department to use any node pool when over-quota is enabled, even if the administrator has not assigned a quota for a specific node pool in a Project or Department.

Workloads can be submitted using a prioritized list of node pools, the node pool selector picks one node pool at a time (according to the prioritized list) and the designated node pool scheduler instance handles the submission request and tries to match the requested resources within that node pool. If the scheduler cannot find resources to satisfy the submitted workload, the node pool selector will move the request to the next node pool in the prioritized list, if no node pool satisfies the request, the node pool selector will start from the first node pool again until one of the node pools satisfies the request.

Node pools table¶

The Node pools table can be found under Nodes in the Run:ai platform.

The Node pools table lists all the node pools defined in the Run:ai platform and allows you to manage them.

The Node pools table consists of the following columns:

Workloads associated with the node pool¶

Click one of the values in the Workload(s) column, to view the list of workloads and their parameters.

Customizing the table view¶

• Filter - Click ADD FILTER, select the column to filter by, and enter the filter values
• Search - Click SEARCH and type the value to search by
• Sort - Click each column header to sort by
• Column selection - Click COLUMNS and select the columns to display in the table
• Download table - Click MORE and then Click Download as CSV
• Show/Hide details - Click to view additional information on the selected row

Adding a new node pool¶

To create a new node pool:

1. Click +NEW NODE POOL
2. Enter a name for the node pool.
   Node pools names must start with a letter and can only contain lowercase Latin letters, numbers or a hyphen ('-’)

3. Enter the node pool label:
   The node pool controller will use this node-label key-value pair to match nodes into this node pool.

   • Key is the unique identifier of a node label.

     • The key must fit the following regular expression: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?/?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$
     • The administrator can put an automatically preset label such as the nvidia.com/gpu.product that labels the GPU type or any other key from a node label.

   • Value is the value of that label identifier (key). The same key may have different values, in this case, they are
     considered as different labels.

     • Value must fit the following regular expression: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
   • A node pool is defined by a single key-value pair. You must not use different labels that are set on the same node by
     different node pools, this situation may lead to unexpected results.

4. Set the GPU placement strategy:

   • Bin-pack - Place as many workloads as possible in each GPU and node to use fewer resources and maximize GPU and node vacancy.
   • Spread Spread workloads across as many GPUs and nodes as possible to minimize the load and maximize the available resources per workload.
   • GPU workloads are workloads that request both GPU and CPU resources

5. Set the CPU placement strategy:

   • Bin-pack - Place as many workloads as possible in each CPU and node to use fewer resources and maximize CPU and node vacancy.
   • Spread - Spread workloads across as many CPUs and nodes as possible to minimize the load and maximize the available resources per workload.
   • CPU workloads are workloads that request purely CPU resources

6. Click CREATE NODE POOL

Labeling nodes for node-pool grouping:¶

The Infrastructure Administrator can use a preset node label such as the nvidia.com/gpu.product that labels the GPU type, or configure any other node label (e.g. faculty=physics).

To assign a label to nodes you want to group into a node pool, set a node label on each node:

1. Get the list of nodes and their current labels using the following command:

   kubectl get nodes --show-labels
   

2. Annotate a specific node with a new label using the following command:

   kubectl label node <node-name> <key>=<value>
   

Editing a node pool¶

1. Select the node pool you want to edit
2. Click EDIT
3. Update the node pool and click SAVE

Deleting a node pool¶

1. Select the node pool you want to delete
2. Click DELETE
3. On the dialog, click DELETE to confirm the deletion

Using API¶

Go to the Node pools API reference to view the available actions