Skip to content

GPU-Capable OLIVE Plugin / Domain Configuration

Introduction

With the release of OLIVE 5.5.0, certain plugin capabilities are now able to leverage GPU hardware to enhance the speed of certain operations and algorithms. This allows us to either use advanced technologies that were previously infeasible to deploy without the GPU speed bump, or to sometimes dramatically increase the speed performance of existing technologies.

Supported GPU Hardware, Software, and Drivers

Refer to the OLIVE Hardware Requirements for information on what versions of CUDA and Nvidia drivers are currently supported.

Currently supported GPU plugins

These plugins currently support GPU operation, when configured as outlined above:

All are capable of running on CPU as well, though in some cases (notably asr-end2end-v1 and asr-whisper-v1) at a drastically reduced speed.

GPU Configuration Requirements

In order to use GPUs with an OLIVE software instance, three things need to happen;

  1. The hardware OLIVE is being installed and run on must have a compatible GPU or GPUs installed. Currently, OLIVE can only use Nvidia GPUs with CUDA cores and properly installed Nvidia drivers. If applicable, you will also need the Nvidia Docker toolkit. Refer to Nvidia's documentation for installation of these.
  2. OLIVE itself must be configured and launched so that it has access to these GPUs. Refer to the documentation specific to your delivery type for information on how to do this. The most likely appropriate reference for this is the Martini Startup Instructions.
  3. Each domain of each plugin that you wish to run on the GPU must be configured to specify this information. This is covered below.

Plugin Domain Configuration for GPU Use

By default, each OLIVE plugin domain is typically configured internally to run solely on the CPU. OLIVE 5.6.0 introduces a new, centralized, more convenient method for configuring the device each domain will use for processing.

CPU/GPU device configuration is now accomplished in an OLIVE Configuration file, called olive.conf, that can be found if a default is provided with your delivery or created if one is not in:

    .../olive5.7.0/oliveAppData/

The format of this file follows:

olive.conf Generic
[device.<plugin name 1>]
<domain 1> = <desired device id>
<domain 2> = <desired device id>
...
<domain N> = <desired device id>

[device.<plugin name 2>]
<domain 1> = <desired device id>
<domain 2> = <desired device id>
...
<domain N> = <desired device id>

Device assignments are made on a per domain basis, not on a plugin-basis. So each domain must be configured separately if wishing to override the default device being used. By default they will still run on CPU, which for most GPU-capable plugins will run significantly more slowly.

Each domain can be independently assigned a device; just because one domain of a plugin is using a GPU doesn't mean the others automatically will or need to. By extension, it is not necessary for all domains of a plugin to run on the same GPU."

Having this device assignment at the domain level allows the distribution of domains across multiple GPUs. For example, if multiple GPUs are available on a system, lower-memory-usage plugins like SAD, SID, and LID may share a single GPU, while heavier plugins like ASR or MT can be flexible enough to assign different language domains of each plugin across multiple GPUs, spreading the memory load to minimize the chance of exhausting GPU memory, while also saving lost time frequently loading and unloading models. The ideal configuration may vary greatly depending on specific customer use case, hardware, and mission needs.

As many or few plugins and domains as desired can be re-assigned in this config file. It is not necessary to include all available plugins in this file; any omitted will use the default, internally assigned device (most often cpu).

Multiple domains can be assigned to the same GPU device, as long as that device's available resources can support each domain.

The available "device IDs" will vary from system to system depending on how many GPUs are available and how they are exposed. Only GPU device(s) that are properly exposed to the server by configuring/launching the server appropriately In general, the valid options are:

  • cpu
  • gpu0
  • gpu1
  • ...
  • gpuN

Many/most systems will only have a single GPU, exposed as gpu0 - the digit required appended to gpu is the device ID reported by Nvidia with the as reported by NVIDIA System Management Interface, or nvidia-smi tool (more info here.

nvidia-smi Example Output (Click to expand)
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.46       Driver Version: 495.46       CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:04:00.0 Off |                  N/A |
| 20%   28C    P0    71W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 20%   28C    P0    73W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:08:00.0 Off |                  N/A |
| 19%   28C    P0    73W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:09:00.0 Off |                  N/A |
| 19%   28C    P0    73W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  Off  | 00000000:84:00.0 Off |                  N/A |
| 18%   26C    P0    71W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  Off  | 00000000:85:00.0 Off |                  N/A |
| 19%   27C    P0    71W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  Off  | 00000000:88:00.0 Off |                  N/A |
| 18%   28C    P0    71W / 250W |      0MiB / 12212MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  Off  | 00000000:89:00.0 Off |                  N/A |
| 17%   27C    P0    73W / 250W |      0MiB / 12212MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

An example of what this config file could look like filled in:

olive.conf Example
[device.asr-end2end-v1.2.0]
english-augmented-v1 = gpu0
french-v1 = cpu
korean-augmented-v1 = gpu1

[device.sad-dnn-v8.1.0]
multi-v1 = gpu0

OLIVE GPU Restrictions and Configuration Notes

GPU Compute Mode: "Default" vs "Exclusive"

The OLIVE software currently assumes that any available GPUs are in the "default" mode. In testing, some configurations of number of OLIVE server workers have been found to cause unexpected issues if the GPUs are configured to be running in "Exclusive" mode. If possible, please configure GPUs that OLIVE will be using in "Default" mode, and if this is not possible, please ensure that the number of workers for the GPU-enabled server is specified to be 1. This is configured in the provided docker-compose.yml file.

To check the mode of your GPU, you can view the Compute Mode field of the nvidia-smi command output. Please refer to Nvidia's instructions for Nvidia Control Panel or the Usage instructions for nvidia-smi for more information on how to set these modes.