OLIVE API Message Protocol Documentation
olive.proto Protocol Buffer Definitions
The messages defined on this page are what define the OLIVE Enterprise API. These messages are how a client application will interact with and provide tasks to an OLIVE Server. The format used by the OLIVE API is based on the Google Protocol Buffer.
For more information on how to integrate the ability to send and receive these messages into a client application using a provided Client API implementation from SRI, refer to the Integrating the (Java) Client API Guide.
For more information on creating your own reference implementation with the functionality of these messages, head over to the Creating an API Reference Implementation page that offers guidelines and information.
Server Management Messages
GetActiveRequest
Message to request the list of ScenicMessages that are still active
GetActiveResult
Response to GetActiveRequest containing the ScenicMessages that are still active
| Field | Type | Label | Description |
|---|---|---|---|
| message_id | string | repeated | List containing the IDs of each message still being processed on the server |
| total_num | string | required | Total number of messages still being processed |
GetStatusRequest
Request a simple server status message, similar to a heartbeat except the request reply is sent on the status port and is requested by the client
GetStatusResult
The result of a GetStatusRequest
| Field | Type | Label | Description |
|---|---|---|---|
| num_pending | uint32 | required | The number of pending jobs |
| num_busy | uint32 | required | the number of active jobs |
| num_finished | uint32 | required | The number of finished jobs |
Heartbeat
A heartbeat Message, acknowledging that the server is running, this message is continuously broadcast by the server on it's status port (this is the only message sent on the status port)
| Field | Type | Label | Description |
|---|---|---|---|
| stats | ServerStats | optional | The current status of the server, optional since status is not available when the server first starts, but a heartbeat is still sent |
| logdir | string | required | The location where the server writes it log files |
ServerStats
Current status of the OLIVE server, sent as part of a Heatbeat message.
| Field | Type | Label | Description |
|---|---|---|---|
| cpu_percent | float | required | The current percentage of CPU used |
| cpu_average | float | required | The average CPU percentage used since the server was started |
| mem_percent | float | required | The percentage of memory used |
| max_mem_percent | float | required | The most memory used since the server was started |
| swap_percent | float | required | The current swap used |
| max_swap_percent | float | required | The max (most) swap space used since the server was started |
| pool_busy | uint32 | required | The number of jobs currently running on the server |
| pool_pending | uint32 | required | The number of jobs queued on the server |
| pool_finished | uint32 | required | The number of jobs completed |
| pool_reinit | bool | required | The number of jobs that need to be re-ran |
| max_num_jobs | uint32 | optional | The max number of concurrent jobs |
| server_version | string | optional | The current version of the server |
LoadPluginDomainRequest
Request a plugin be pre-loaded to optimize later score request(s)
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
LoadPluginDomainResult
Acknowledgment that a plugin is being loaded
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | True if the request is being loaded (but receipt of this message does not guarantee the plugin has finished loaded) |
RemovePluginDomainRequest
Unload (remove from memory) a previously loaded plugin. Use to free resources on the server or force reloading of a plugin/domain
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to remove |
| domain | string | optional | The domain to remove, if omitted all domains removed for this plugin. |
RemovePluginDomainResult
Acknowledgment that a plugin/domain has been removed (unloaded)
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | True if the plugin was been removed |
PluginDirectoryRequest
Use a PluginDirectoryRequest message to receive the list of plugins available on the server. A Plugin performs tasks such as SAD, LID, SID, or KWS. There may be multiple plugins registered for a given task. A plugin typically has one or more Domains. Plugins contain the code of the recognizer, while Domains correspond to a particular training or adaptation sessions. Domains therefore represent the data/conditions. A plugin and domain together are necessary to perform scoring.
PluginDirectoryResult
The collection of plugins available on the server, response to PluginDirectoryRequest
| Field | Type | Label | Description |
|---|---|---|---|
| plugins | Plugin | repeated | The available plugins |
Global Scorer Messages
GlobalScorer is an OLIVE Plugin Trait defining a/the type of scoring the plugin is capable of. For more details about what it means to be a GlobalScorer, refer to the relevant section of the Plugin Traits page.
For an example of the code that would be used to build and submit a GlobalScorer request, refer to the relevant section of the Integrating the (Java) Client API Guide.
GlobalScore
The global score for a class
| Field | Type | Label | Description |
|---|---|---|---|
| class_id | string | required | The class |
| score | float | required | The score associated with the class |
| confidence | float | optional | An optional confidence value when part of a calibration report |
| comment | string | optional | An optional suggested action when part of a calibration report |
GlobalScorerRequest
Request global scoring using the specified plugin. The plugin must implement the GlobalScorer trait to handle this request. If this request is successful, then one set of scores is returned since the audio submission is assumed to be mono. If submitting multichannel audio then the audio is merged (unless a channel specified in the Audio message, then that channel is used) to produce one set of global scores.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to invoke |
| domain | string | required | The domain |
| audio | Audio | optional | The audio to analyze/score. Either audio or vector must be set. |
| vector | AudioVector | optional | The preprocessed audio vector to analyze/score. Either audio or vector must be set. |
| option | OptionValue | repeated | Any options specified |
| class_id | string | repeated | Optionally specify the classes to be scored |
GlobalScorerResult
The result from a GlobalScorerRequest, having zero or more GlobalScore elements
| Field | Type | Label | Description |
|---|---|---|---|
| score | GlobalScore | repeated | The class scores |
Region Scorer Messages
RegionScorer is an OLIVE Plugin Trait defining a/the type of scoring the plugin is capable of. For more details about what it means to be a RegionScorer, refer to the relevant section of the Plugin Traits page.
For an example of the code that would be used to build and submit a RegionScorer request, refer to the relevant section of the Integrating the (Java) Client API Guide.
RegionScore
The basic unit of a region score. There may be multiple RegionScore values in a RegionScorerResult
| Field | Type | Label | Description |
|---|---|---|---|
| start_t | int32 | required | Begin-time of the region (in seconds) |
| end_t | int32 | required | End-time of the region (in seconds) |
| class_id | string | required | Class ID associated with region |
| score | float | optional | Optional score associated with the class_id label |
RegionScorerRequest
Request region scoring for the specified plugin/domain. The plugin must implement the RegionScorer trait to handle this request. If this request is successful, then one set of scores is returned since the audio submission is assumed to be mono. If submitting multichannel audio then the audio is merged (unless a channel specified in the Audio message, then that channel is used) to produce one set of region scores.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to invoke |
| domain | string | required | The domain |
| audio | Audio | required | The audio to analyze/score |
| option | OptionValue | repeated | Any options specified |
| class_id | string | repeated | Optionally specify the classes to be scored |
RegionScorerResult
The set of region score results, response to RegionScorerRequest
| Field | Type | Label | Description |
|---|---|---|---|
| region | RegionScore | repeated | The scored regions |
Frame Scorer Messages
FrameScorer is an OLIVE Plugin Trait defining a/the type of scoring the plugin is capable of. For more details about what it means to be a FrameScorer, refer to the relevant section of the Plugin Traits page.
For an example of the code that would be used to build and submit a FrameScorer request, refer to the relevant section of the Integrating the (Java) Client API Guide.
FrameScorerRequest
Request frame scoring using the specified plugin and audio. The plugin must implement the FrameScorer trait to handle this request. If this request is successful, then one set of scores is returned since the audio submission is assumed to be mono. If submitting multichannel audio then the audio is merged (unless a channel specified in the Audio message, then that channel is used) to produce one set of frame scores.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to invoke |
| domain | string | required | The domain |
| audio | Audio | required | The audio to analyze/score |
| option | OptionValue | repeated | Any options specified |
| class_id | string | repeated | Optionally specify the classes to be scored |
FrameScorerResult
The results from a FrameScorerRequest
| Field | Type | Label | Description |
|---|---|---|---|
| result | FrameScores | repeated | List of frame scores by class_id |
FrameScores
The basic unit of a frame score, returned in a FrameScorerRequest
| Field | Type | Label | Description |
|---|---|---|---|
| class_id | string | required | The class ID to which the frame scores pertain |
| frame_rate | int32 | required | The number of frames per second |
| frame_offset | double | required | The offset to the center of the frame 'window' |
| score | double | repeated | The frame-level scores for the class_id |
Text Transformation Messages
TextTransformer is an OLIVE Plugin Trait for scoring using text (instead of audio). For more details about what it means to be a TextTransformer, refer to the relevant section of the Plugin Traits page.
TextTransformationRequest
Request the transformation of a text/string using MT
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to invoke |
| domain | string | required | The domain |
| text | string | optional | The string text to analyze/score, Optional as of OLIVE 5.0 since data input(s) can be specified as part of a workflow |
| option | OptionValue | repeated | Any options specified |
| class_id | string | repeated | Optionally specify the classes to be scored |
TextTransformationResult
The response to a TextTransformationRequest
| Field | Type | Label | Description |
|---|---|---|---|
| transformation | TextTransformation | repeated |
TextTransformation
The text value returned in a TextTransformationResult
| Field | Type | Label | Description |
|---|---|---|---|
| class_id | string | required | A classifier for this result, usually just 'text' |
| transformed_text | string | required | The text result |
Audio Alignment Messages
AudioAlignmentScorer is an OLIVE Plugin Trait for alignment of two or more audio inputs. For more details about what it means to be a TextTransformer, refer to the relevant section of the Plugin Traits page.
AudioAlignmentScoreRequest
Request the alignment of two or more audio inputs
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to invoke |
| domain | string | required | The domain |
| audios | Audio | repeated | The audio to analyze/score, Optional as of OLIVE 5.0 since Audio can be specified |
| option | OptionValue | repeated | as part of a workflow. If specified there should be two or more audio inputs |
Any options specified | | class_id | string | repeated | Optionally specify the classes to be scored |
AudioAlignmentScoreResult
The result of a AudioAlignmentScoreRequest
| Field | Type | Label | Description |
|---|---|---|---|
| scores | AudioAlignmentScore | repeated |
AudioAlignmentScore
A score in an AudioAlignmentScoreResult
| Field | Type | Label | Description |
|---|---|---|---|
| reference_audio_label | string | required | The source or reference audio name (file 1) |
| other_audio_label | string | required | The name of the audio input in comparison (file 2) |
| shift_offset | float | required | shift offset between the audion in the reference and |
| confidence | float | required | The confidence of this score |
Global Comparer Messages
GlobalComparerReport
The visual representation of a global comparison
| Field | Type | Label | Description |
|---|---|---|---|
| type | ReportType | required | The type of report (normally a PDF) |
| report_data | bytes | required | The serialized report |
GlobalComparerRequest
Request the comparison of two audio submission. The plugin must implement the GlobalComparer trait to handle this request
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to invoke |
| domain | string | required | The domain |
| audio_one | Audio | required | One of two audio submissions to analyze/score. |
| audio_two | Audio | required | One of two audio submissions to analyze/score. |
| option | OptionValue | repeated | Any options specified |
| class_id | string | repeated | Optionally specify the classes to be scored // todo remove! |
GlobalComparerResult
The result of a GlobalComparerRequest
| Field | Type | Label | Description |
|---|---|---|---|
| results | Metadata | repeated | The metadata/scores returned from a global compare analysis |
| report | GlobalComparerReport | repeated | A comparison report generated by the plugin |
ReportType
Possible report formats
| Name | Number | Description |
|---|---|---|
| 1 | ||
| PNG | 2 | |
| GIF | 3 | |
| JPEG | 4 | |
| TIFF | 5 |
Class Modifier Messages
ClassModificationRequest
Request a modification of a class for the specified plugin. The plugin must implement the ClassModifier Trait to handle this request.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
| class_id | string | required | The ID of the class being enrolled/modified |
| addition | Audio | repeated | List of Audio, action pairs to apply to the class |
| removal | Audio | repeated | List of Audio, action pairs to apply to the class |
| addition_vector | AudioVector | repeated | List of preprocessed audio vector to apply to the class |
| finalize | bool | optional | Whether or not to finalize the class. You can send multiple ClassModificationRequests and only finalize on the last request for efficiency. Default: true |
| option | OptionValue | repeated | Any modification options |
ClassModificationResult
Response to ClassModificationRequest.
| Field | Type | Label | Description |
|---|---|---|---|
| addition_result | AudioResult | repeated | Provides feedback about the success/failure of individual audio additions |
| removal_result | AudioResult | repeated | Provides feedback about the success/failure of individual audio removals |
| vector_addition_result | AudioResult | repeated | Provides feedback about the success/failure of individual audio vector additions |
ClassRemovalRequest
Request removal of a class in the specified plugin/domain
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
| class_id | string | required | The id of the class to be removed |
ClassRemovalResult
Acknowledgment that a ClassRemovalRequest was received
AudioResult
The feedback/description of class modification for a result in a ClassModificationResult message
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | Whether or not the individual audio succeeded |
| message | string | optional | Description of what occurred on this audio |
Audio Converter Messages
AudioModification
The contents of an AudioModificationResult.
| Field | Type | Label | Description |
|---|---|---|---|
| audio | AudioBuffer | required | The transformed audio |
| message | string | required | Description of how this audio was transformed, or an error description. Not sure if needed? |
| scores | Metadata | repeated | Zero or more scores (metadata) about the modifed audio. Metadata is a list of name/value pairs. |
AudioModificationRequest
Request enhancement (modification) of the submitted audio. The plugin must support the AudioConverter trait to support this request
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
| requested_channels | uint32 | required | Convert audio to have this number of channels |
| requested_rate | uint32 | required | Convert audio to this sample rate |
| modifications | Audio | repeated | List of Audio, action pairs to apply to the class - may have to limit to one audio submission per request, not sure how to handle multiple results |
| option | OptionValue | repeated | Any options specified |
AudioModificationResult
The result of an AudioModificationRequest
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | Whether or not the individual audio modification succeeded |
| modification_result | AudioModification | repeated | Provides feedback about the success/failure of individual audio additions. |
Audio Vectorizer Messages
AudioVector
Represents audio preprocessed by a plugin/domain.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The origin plugin |
| domain | string | required | The origin domain |
| data | bytes | required | The audio vector data, varies by plugin |
| params | Metadata | repeated | Name/value parameter data generated by the plugin and needed for later processing |
PluginAudioVectorRequest
Request one or more audio submissions be vectorized (preprocesssed) by the specified plugin. The resulting vectorized audio can only be processed by the same plugin. A plugin must support the AudioVectorizer Trait to support this request.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
| addition | Audio | repeated | List of Audio to process |
PluginAudioVectorResult
The result of a PluginAudioVectorRequest, containing a set of VectorResults
| Field | Type | Label | Description |
|---|---|---|---|
| vector_result | VectorResult | repeated | The results of processing the submitted audio. One result per audio addition. |
VectorResult
The status of the vector request, and if successful includes an AudioVector
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | Whether or not the audio was successfully processed |
| message | string | optional | Description of what occurred to cause an error |
| audio_vector | AudioVector | optional | If successful, the vectorized audio |
ClassExportRequest
Exports an existing class enrollment (i.e. speaker enrollment) from the server for the specified class_id. The plugin must support the ClassExporter trait to handle this request.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
| class_id | string | required | The ID of the class model to export |
ClassExportResult
The result of an enrollment export
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | Whether or not the individual audio succeeded |
| message | string | optional | Description of what occurred to cause an error |
| enrollment | EnrollmentModel | optional | If successful, then this is the exported model for the specified class. |
ClassImportRequest
Used to import an enrollment model (exported via a ClassExportRequest). Only plugins that support the ClassExporter trait can handle this request. Only import an enrollment into the same plugin AND domain as previously exported.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
| class_id | string | optional | Import the model using this class name, instead of the original name |
| enrollment | EnrollmentModel | required | the enrollment to import |
ClassImportResult
The status of a ClassImportRequest.
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | Whether or not the import succeeded |
| message | string | optional | Description of what occurred to cause an error |
EnrollmentModel
An enrollment model for a specific plugin and domain. This is used to save a current enrollment or restore a class enrollment via a ClassImportRequest. This model is not used as an AudioVector in a scoring requests.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The origin plugin |
| domain | string | required | The origin domain |
| class_id | string | required | the class_id of the enrollment |
| data | bytes | required | The enrollment model data |
| params | Metadata | repeated | Name/value parameter data generated by the plugin and needed for later processing |
Updater Messages
ApplyUpdateRequest
Used to request an update of a Plugin that supports the Update trait. Use GetUpdateStatusRequest to check if a plugin is ready for an update, otherwise this request is ignored by the server/plugin
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to apply the update |
| domain | string | required | The domain |
| params | Metadata | repeated | Name/value options, plugin dependent |
ApplyUpdateResult
This message is immediately returned after an ApplyUpdateRequest, as the updating process can take an extended time range to complete.
| Field | Type | Label | Description |
|---|---|---|---|
| successful | bool | required | True if the plugin is being updated |
GetUpdateStatusRequest
Used to request the status for a Plugin that supports the Update trait
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
GetUpdateStatusResult
The result of a GetUpdateStatusRequest message.
| Field | Type | Label | Description |
|---|---|---|---|
| update_ready | bool | required | True if the plugin has determiend it is ready for an update |
| last_update | DateTime | optional | The date of the last update, if any |
| params | Metadata | repeated | Zero or Metadata values describing the update status of the plugin. Metadata is a list of name/value pairs. |
DateTime
Date and time info
| Field | Type | Label | Description |
|---|---|---|---|
| year | uint32 | required | Year |
| month | uint32 | required | Month |
| day | uint32 | required | Day |
| hour | uint32 | required | Hour |
| min | uint32 | required | Minute |
| sec | uint32 | required | Seconds |
Learning Trait Messages
These messages are used by plugins that support adaptation and/or training.
PreprocessAudioAdaptRequest
Request preprocessing of this audio submission, which may be part of an adaptation set. Adaptation can be unsupervised (neither class_id, start_t, and end_t set) or supervised by setting class_id or class_id, start_t, and end_t. Adaptation should be finalized by calling either SupervisedAdaptationRequest or UnsupervisedAdaptationRequest. Plugins must support either the SupervisedAdapter or UnsupervisedAdapter trait to handle this request.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin |
| domain | string | required | The domain |
| audio | Audio | required | The submitted audio |
| adapt_space | string | required | A unique name for the client where pre-processed data is stored |
| class_id | string | optional | The id of the class annotation being preprocessed (supervised training) |
| start_t | uint32 | optional | Begin-time of the region (in seconds) |
| end_t | uint32 | optional | End-time of the region (in seconds) |
PreprocessAudioAdaptResult
The result of a PreprocessAudioAdaptRequest
| Field | Type | Label | Description |
|---|---|---|---|
| audio_id | string | required | The ID of the preprocessed audio |
| duration | double | required | The duration of the audio |
Supervised Adapter Messages
SupervisedAdaptationRequest
Finalize adaptation of the specified plugin/domain using audio preprocessed using calls to PreprocessAudioAdaptRequest. The plugin must implement the SupervisedAdapter trait to handle this request.
When you adapt or train, you are creating a new domain for the target plugin, that is based on the domain passed in to the 'domain' field of this call. This new domain is specific to a plugin, so it is created within the plugin, and will be named with the string passed to SupervisedAdaptationRequest as 'new_domain'.
To actually use this new domain, future scoring or enrollment requests must specify this new domain name, instead of the original, using the value specified during adaptation as the 'new_domain' field. To have access to this new domain, either restart the server, or send a RemovePluginDomainRequest message to the server, which will force a reload of that plugin.
Upon successful completion of SupervisedAdaptationRequest, a SupervisedAdaptationResult message should be received, containing the path to the newly created domain on the server's file system.
The file sizes of the actual models will not change as a result of adaptation. Rather, the values stored inside these files will.
| Field | Type | Label | Description |
|---|---|---|---|
| plugin | string | required | The plugin to invoke |
| domain | string | required | The domain to adapt |
| new_domain | string | required | the new domain name |
| class_annotations | ClassAnnotation | repeated | The annotations to use for adaptation, audio annotations are created via PreprocessAudioAdaptRequest calls |
| adapt_space | string | required | A unique name for the client where pre-processed data is stored |
SupervisedAdaptationResult
Acknowledgment message that adaptation successfully completed. Informs the client of the full path of the new domain created by the SupervisedAdaptationRequest.
| Field | Type | Label | Description |
|---|---|---|---|
| new_domain | string | required | Confirmation of the new domain name |
ClassAnnotation
Set of annotations for a class
| Field | Type | Label | Description |
|---|---|---|---|
| class_id | string | required | The class ID (such as speaker name or language name) |
| annotations | AudioAnnotation | repeated | the set of all audio annotations for this class. |
AudioAnnotation
A set of audio annotations for a specific audio submission
| Field | Type | Label | Description |
|---|---|---|---|
| audio_id | string | required | The audio ID returned in a PreprocessAudioAdaptResult or PreprocessAudioTrainResult message |
| regions | AnnotationRegion | repeated | Set of annotations |
Workflow Messages
Not yet defined
Basic Types
Trait
A Trait implemented by a plugin
| Field | Type | Label | Description |
|---|---|---|---|
| type | TraitType | optional | The trait type |
| options | OptionDefinition | repeated | Any options specific to this plugin's implementation of the trait |
Plugin
The description of a plugin
| Field | Type | Label | Description |
|---|---|---|---|
| id | string | optional | The id of the plugin |
| task | string | optional | e.g. LID, SID, SAD, KWS, AED, etc. |
| label | string | optional | Display label for plugin |
| desc | string | optional | A brief description of how the plugin works/technologies it employs. |
| vendor | string | optional | A brief description of how the plugin works/technologies it employs. |
| domain | Domain | repeated | The domains owned by this plugin |
| trait | Trait | repeated | The traits (capabilities) of this plugin |
Domain
A description of a domain.
| Field | Type | Label | Description |
|---|---|---|---|
| id | string | optional | The ID of the domain |
| label | string | optional | Display label for the domain |
| desc | string | optional | A brief description of the domain conditions |
| class_id | string | repeated | The list of classes known to this domain |
Envelope
Every message passed between the server and client is an instance of Envelope. An Envelope can contain multiple ScenicMessage instances, so it's important to iterate through them all when you receive an Envelope.
| Field | Type | Label | Description |
|---|---|---|---|
| message | ScenicMessage | repeated | The messages to be sent |
| sender_id | string | required | string description of the message sender |
ScenicMessage
A ScenicMessage represents a single logical message between a client and server. It is placed within an Envelope. It contains nested messages in serialized form. The message_type field is used to determine the type of the nested data. Not all ScenicMessage instance will have message_data, and some may have multiple, but they will all be of the same type. It depends on the value of message_type.
| Field | Type | Label | Description |
|---|---|---|---|
| message_id | string | required | id issued by client (and unique to that client) used to track a request. Any reply for that request will have the same id. |
| message_type | MessageType | required | type of message |
| message_data | bytes | repeated | nested message data that can be deserialized according to message_type. Some messages do not have nested data, some have multiple records |
| error | string | optional | error message; if present an error has occurred on the server |
| info | string | optional | informational message, typically used to explain why message_data is empty but no error is reported |
MessageType
The MessageType enum provides a value for each top-level SCENIC message. It is used within a ScenicMessage to indicate the type of the serialized message contained therein.
| Name | Number | Description |
|---|---|---|
| PLUGIN_DIRECTORY_REQUEST | 1 | |
| PLUGIN_DIRECTORY_RESULT | 2 | |
| GLOBAL_SCORER_REQUEST | 3 | |
| GLOBAL_SCORER_RESULT | 4 | |
| REGION_SCORER_REQUEST | 5 | |
| REGION_SCORER_RESULT | 6 | |
| FRAME_SCORER_REQUEST | 7 | |
| FRAME_SCORER_RESULT | 8 | |
| CLASS_MODIFICATION_REQUEST | 9 | |
| CLASS_MODIFICATION_RESULT | 10 | |
| CLASS_REMOVAL_REQUEST | 11 | |
| CLASS_REMOVAL_RESULT | 12 | |
| GET_ACTIVE_REQUEST | 13 | |
| GET_ACTIVE_RESULT | 14 | |
| LOAD_PLUGIN_REQUEST | 15 | |
| LOAD_PLUGIN_RESULT | 16 | |
| GET_STATUS_REQUEST | 17 | |
| GET_STATUS_RESULT | 18 | |
| HEARTBEAT | 19 | |
| PREPROCESS_AUDIO_TRAIN_REQUEST | 20 | |
| PREPROCESS_AUDIO_TRAIN_RESULT | 21 | |
| PREPROCESS_AUDIO_ADAPT_REQUEST | 22 | |
| PREPROCESS_AUDIO_ADAPT_RESULT | 23 | |
| SUPERVISED_TRAINING_REQUEST | 24 | |
| SUPERVISED_TRAINING_RESULT | 25 | |
| SUPERVISED_ADAPTATION_REQUEST | 26 | |
| SUPERVISED_ADAPTATION_RESULT | 27 | |
| UNSUPERVISED_ADAPTATION_REQUEST | 28 | |
| UNSUPERVISED_ADAPTATION_RESULT | 29 | |
| CLASS_ANNOTATION | 30 | |
| AUDIO_ANNOTATION | 31 | |
| ANNOTATION_REGION | 32 | |
| REMOVE_PLUGIN_REQUEST | 33 | |
| REMOVE_PLUGIN_RESULT | 34 | |
| AUDIO_MODIFICATION_REQUEST | 35 | |
| AUDIO_MODIFICATION_RESULT | 36 | |
| PLUGIN_AUDIO_VECTOR_REQUEST | 37 | |
| PLUGIN_AUDIO_VECTOR_RESULT | 38 | |
| CLASS_EXPORT_REQUEST | 39 | |
| CLASS_EXPORT_RESULT | 40 | |
| CLASS_IMPORT_REQUEST | 41 | |
| CLASS_IMPORT_RESULT | 42 | |
| APPLY_UPDATE_REQUEST | 43 | |
| APPLY_UPDATE_RESULT | 44 | |
| GET_UPDATE_STATUS_REQUEST | 45 | |
| GET_UPDATE_STATUS_RESULT | 46 | |
| GLOBAL_COMPARER_REQUEST | 47 | |
| GLOBAL_COMPARER_RESULT | 48 |
| AUDIO_ALIGN_REQUEST | 68 | | | AUDIO_ALIGN_RESULT | 69 | | | TEXT_TRANSFORM_REQUEST | 70 | | | TEXT_TRANSFORM_RESULT | 71 | | | PREPROCESSED_AUDIO_RESULT | 72 | | | INVALID_MESSAGE | 73 | |
OptionType
Classifies how a OptionDefinition (TraitOption) should be represented in a UI widget
| Name | Number | Description |
|---|---|---|
| BOOLEAN | 1 | |
| CHOICE | 2 |
TraitType
The list of possible traits that a plugin can implement
| Name | Number | Description |
|---|---|---|
| GLOBAL_SCORER | 1 | |
| REGION_SCORER | 2 | |
| FRAME_SCORER | 3 | |
| CLASS_ENROLLER | 4 | |
| CLASS_MODIFIER | 5 | |
| SUPERVISED_TRAINER | 6 | |
| SUPERVISED_ADAPTER | 7 | |
| UNSUPERVISED_ADAPTER | 8 | |
| AUDIO_CONVERTER | 9 | |
| AUDIO_VECTORIZER | 10 | |
| CLASS_EXPORTER | 11 | |
| UPDATER | 12 | |
| LEARNING_TRAIT | 13 | |
| GLOBAL_COMPARER | 14 | |
| TEXT_TRANSFORMER | 15 | |
| AUDIO_ALIGNMENT_SCORER | 16 |
TaskType
| Name | Number | Description |
|---|---|---|
| SAD | 1 | Speech Activity Detection |
| SID | 2 | Speaker ID |
| SDD | 3 | Speaker ID, but output in regions |
| LID | 4 | Language ID |
| LDD | 5 | Language ID, but output in regions |
| KWS | 6 | Keyword Spotting |
| TPD | 7 | Topic Detection |
| VTD | 8 | Voice Type Discrimination |
| GID | 9 | Gender ID |
| GDD | 10 | Gender ID, but output in regions |
| ASR | 11 | Automatic Speech Recognition |
| ENH | 12 | Audio Enhancement |
| CMP | 13 | Audio Comparison |
| SDD | 14 | Speaker Detection |
| DIA | 15 | Diarization |
| QBE | 16 | Query by Example |
| SHL | 17 | Speaker Highlighing |
| FID | 18 | Face ID |
| TMT | 19 | Text Machine Translation |
| ALN | 20 | Audio Alignment |
OptionDefinition
A plugin TraitOption, describing how a plugin trait is used
| Field | Type | Label | Description |
|---|---|---|---|
| name | string | required | The name/ID of the option |
| label | string | required | Display label for the option |
| desc | string | optional | A description of the option |
| type | OptionType | required | The type of the option (boolean, choice/drop-down, etc) |
| choice | string | repeated | Optional list of choices used by CHOICE type options |
| default | string | optional | The default option in the list of Options |
OptionValue
A name/value property pair
| Field | Type | Label | Description |
|---|---|---|---|
| name | string | required | The name/ID of the option |
| value | string | required | The option value as a string |
InputDataType
Workflow(?) Input Data Types
| Name | Number | Description |
|---|---|---|
| AUDIO | 1 | |
| VIDEO | 2 | |
| TEXT | 3 | |
| IMAGE | 4 |
InputType
| Name | Number | Description |
|---|---|---|
| FRAME | 1 | |
| REGION | 2 |
JobClass
| Field | Type | Label | Description |
|---|---|---|---|
| job_name | string | required | The parent job name in a Workflow JobDefinition |
| task | TaskClass | repeated |
TaskClass
| Field | Type | Label | Description |
|---|---|---|---|
| task_name | string | required | The ID from the associated WorkflowTask (consumer_result_label) |
| class_id | string | repeated | Zero or more class IDs available to this task. Some tasks do not support classes |
| class_label | string | optional | An optional label/name to describe the classes used by this task such a 'speaker' or 'language' |
| classes_label | string | optional | The speaker label when refering to plural classes, such as speakers, or languages |
Shared Types
Audio
Represents an audio. Can either refer to a local file or embed an audio buffer directly. The path and audioSamples fields should be treated as mutually exclusive, with one and only one of these fields implemented
For more information regarding what types of audio are currently supported by the OLIVE server, see the Supported Audio Formats page.
| Field | Type | Label | Description |
|---|---|---|---|
| path | string | optional | Path to the audio file represented by this record (if not specified then audio is input as a buffer) |
| audioSamples | AudioBuffer | optional | Audio included as a buffer (if not specified, then path must be set) |
| selected_channel | uint32 | optional | Optional - if using multi-channel audio and 'mode' is SELECTED, then this channel is provided to the plugin(s). |
| regions | AnnotationRegion | repeated | Optional annotated regions for this audio |
| label | string | optional | Optional - label used to identify this audio input |
AnnotationRegion
A single pair of timestamps (start and end) that make up an annotated region. Timestamps are in seconds.
| Field | Type | Label | Description |
|---|---|---|---|
| start_t | uint32 | required | Begin-time of the region (in seconds) |
| end_t | uint32 | required | End-time of the region (in seconds) |
AudioBuffer
Audio is contained in a buffer (and the path filed is NOT set in Audio) - by default the audio in the buffer should be PCM encoded, unless the buffer contains a serialized file (unencoded) in which case the serialized_file must be set to true. If the data has been decoded and is not PCM encoded data, then the encoding field must be specified
For more information regarding what types of audio are currently supported by the OLIVE server, see the Supported Audio Formats page.
| Field | Type | Label | Description |
|---|---|---|---|
| channels | uint32 | optional | The number of channels contained in data, ignored for serialized buffers |
| samples | uint32 | optional | The number of samples (in each channel), ignored for serialized buffers |
| rate | uint32 | optional | The sample rate, ignored for serialized buffers |
| bit_depth | AudioBitDepth | optional | The number of bits in each sample, ignored for serialized buffers |
| data | bytes | required | Should be channels * samples long, striped by channels |
| serialized_file | bool | optional | Optional - true if data contains a serialized buffer |
| encoding | AudioEncodingType | optional | Optional - Not Yet supported - the audio encoding type. Assumed to be PCM if not specified. Ignored for serialized buffer |
AudioEncodingType
Audio encoding types
| Name | Number | Description |
|---|---|---|
| PCMU8 | 1 | |
| PCMS8 | 2 | |
| PCM16 | 3 | |
| PCM24 | 4 | |
| PCM32 | 5 | |
| FLOAT32 | 6 | |
| FLOAT64 | 7 | |
| ULAW | 8 | |
| ALAW | 9 | |
| IMA_ADPCM | 10 | |
| MS_ADPCM | 11 | |
| GSM610 | 12 | |
| G723_24 | 13 | |
| G721_32 | 14 | |
| DWW12 | 15 | |
| DWW16 | 16 | |
| DWW24 | 17 | |
| VORBIS | 18 | |
| VOX_ADPCM | 19 | |
| DPCM16 | 20 | |
| DPCM8 | 21 |
AudioBitDepth
Number of bits in each audio sample
| Name | Number | Description |
|---|---|---|
| BIT_DEPTH_8 | 1 | |
| BIT_DEPTH_16 | 2 | |
| BIT_DEPTH_24 | 3 | |
| BIT_DEPTH_32 | 4 |
Metadata
The parent container for Metadata so that typed name/value properties can be transported in a generalized way
| Field | Type | Label | Description |
|---|---|---|---|
| type | MetadataType | required | Indicates the type of this metadata, so it can be deserialized to the appropriate type |
| name | string | required | The name (key) for this metadata |
| value | bytes | required | The value is one of MetadataType, must be deserialized by the client into the type specified by type |
MetadataType
Data types supported in an AudioModificationResult's metadata:
| Name | Number | Description |
|---|---|---|
| STRING_META | 1 | |
| INTEGER_META | 2 | |
| DOUBLE_META | 3 | |
| BOOLEAN_META | 4 | |
| LIST_META | 5 |
BooleanMetadata
Value as boolean
| Field | Type | Label | Description |
|---|---|---|---|
| value | bool | required |
DoubleMetadata
Value as a double
| Field | Type | Label | Description |
|---|---|---|---|
| value | double | required |
IntegerMetadata
Value as an integer
| Field | Type | Label | Description |
|---|---|---|---|
| value | int32 | required |
ListMetadata
Value as list of Metadata values
| Field | Type | Label | Description |
|---|---|---|---|
| type | MetadataType | repeated | The type for the corresponding element |
| value | bytes | repeated | The value is one or more MetadataType elements, each element must be deserialized by the client into the type specified by type. For example, for the type, STRING_META, deserialize data as StringMetadata |
StringMetadata
Value as a string
| Field | Type | Label | Description |
|---|---|---|---|
| value | string | required |