lid-embedpldaSmolive-v1 (Language Identification)
Version Changelog
| Plugin Version | Change |
|---|---|
| v1.0.0 | Initial plugin release with OLIVE 5.4.0 - This plugin is based off of lid-embedplda-v3 but features model quantization and pruning in addition to a full-resolution model for hardware compatibility when the pruned model cannot operate. |
Description
This plugin is based heavily on its predecessor, lid-embedplda-v3, with the important distinction that the model has been modified with both quantization and pruning. Quantized models perform computations at a reduced bit resolution (integer8 in our case) than the standard floating point precision allowing for a more compact model representation and the use of high performance vectorized operations on many hardware platforms. Further details can be found at pytorch Pruning aims to make neural network models smaller by removing a subset of the nodes that have minimal impact on the performance. The pruned neural network is fine-tuned on the target task to minimize the performance loss. The result is a much lighter-weight and often faster-performing model that sacrifices little-to-no accuracy.
LID plugins analyze an audio segment to produce a detection score for each of the enabled language or dialect classes for the domain in use. A plugin domain could consist of 50 or more languages and dialects in a single plugin, or as few as one for use cases where the customer is only focused on a single target class. Some plugin domains are solely focused on dialect or sub-language recognition, such as languages of China. Several LID plugins allow users to add new classes or augment existing classes with more data for the class to improve accuracy.
Language recognition plugin for clean telephone or microphone data, based on a language embeddings DNN fed with acoustic DNN bottleneck features, and language classification using a PLDA backend and multi-class calibration. This plug-in has been configured to allow enrollment for the augmentation of existing classes and the addition of new classes. The backend and calibration modules are rapidly retrained with the addition of enrollment data.
Domains
- multi-int8-v1
- Generic domain for most close talking conditions with signal-to-noise ratio above 10 dB. Currently set up with 16 languages configured (optionally configurable to up to 54 languages). See below for the currently-configured and available languages. See the configuring languages section for instructions on reconfiguring the available languages if necessary.
Inputs
Audio file or buffer and an optional identifier.
Outputs
Generally, a list of scores for all classes in the domain, for the entire segment. As with SAD and SID, scores are generally log-likelihood ratios where a score of greater than “0” is considered a detection. Plugins may be altered to return only detections, rather than a list of classes and scores, but this is generally done on the client side for sake of flexibility.
An example output excerpt:
input-audio.wav Amharic -6.73527861
input-audio.wav Arabic -3.31796265
input-audio.wav English 8.22701168
input-audio.wav French -2.98071671
input-audio.wav Iranian Persian -5.55558729
input-audio.wav Japanese -6.01283073
input-audio.wav Korean -5.64162636
input-audio.wav Mandarin -4.81163836
input-audio.wav Portuguese -1.93523705
input-audio.wav Russian -5.60199690
input-audio.wav Spanish -3.70800495
input-audio.wav Tagalog -4.86510944
input-audio.wav Vietnamese -5.10995102
Functionality (Traits)
The functions of this plugin are defined by its Traits and implemented API messages. A list of these Traits is below, along with the corresponding API messages for each. Click the message name below to go to additional implementation details below.
- GLOBAL_SCORER – Score all submitted audio, returning a single score for the entire audio segment for each of the enrolled and enabled languages of interest.
- CLASS_MODIFIER – Enroll new language models or replace/augment existing language models with additional data.
Compatibility
OLIVE 5.4+
Limitations
Known or potential limitations of the plugin are outlined below.
All current LID plugins assume that an audio segment contains only a single language and may be scored as a unit. If a segment contains multiple languages the entire segment will still be scored as a unit. In many cases, a minimum duration of speech of 2 seconds is required in order to output scores. This value can optionally be overwritten, but scores provided for such short segments will be volatile.
Quantized Model Hardware Compatibility
There are certain host/hardware requirements for the quantized models to be able to run; namely support for avx2. To avoid the situation where a lack of this support would cause the plugin to become nonfuntional, a full-bit (float32) model has been included that will be loaded and used in the rare case that the quantized model fails to load. This will use more memory, but allow the plugin to function.
Minimum Speech Duration
The system will only attempt to perform language identification if the submitted audio segment contains more than 2 seconds of detected speech.
Languages of Low Confidence
Many of the language models that are included and hidden within the domain's data model, disabled by default, do not contain enough data within the model for reliable detection of this language, and are included solely to help with score calibration, and differentiating other languages. If in doubt regarding whether an enrolled language should be used for detection or not, please reach out to SRI for clarification.
Comments
Language/Dialect Detection Granularity
LID plugins attempt to distinguish dialects (ie., Tunisian Arabic and Levantine Arabic) or a base language class (such as Arabic). These can be mapped back to the base language if desired. This requires one change to be enabled.
- A mapping file 'dialect_language.map' must exist within the domain of the plugin for which mapping is to be performed (eg. domains/multi-v1/dialect_language.map). This file is a tab-delimited, two-column file that lists each mapping for the dialect to the languages as "
\t ". Example lines include: Levantine Arabic Arabic Tunisian Arabic Arabic
In the example above, the output labels of the dialects will be mapped to the same base language 'Arabic'. Note the exception in which mapping is not performed is for user-enrolled languages where it is assumed the user has provided the dialect or language label based on their requirements.
Note that we recommend users request these mapping files from SRI, or request the mapping to be performed before delivery of the plugin so that SRI can test and validate the final mapping before delivery.
Note also that the system will not allow you to create an enrollment with the same class name that you have languages mapped to. This is to avoid confusing situations where the system isn't sure if it should be considering the original pre-mapped models, or the newly enrolled user model. You must provide a unique name for any new language enrollments, that does not conflict with the dialect_langage.map. If you have already enrolled a conflicting model, and then add a mapping to this same name, the plugin will provide a warning message and intentionally fail to load.
Enrollments
Some recent LID plugins allows class modifications. A class modification is essentially an enrollment capability similar to SID. A new enrollment is created with the first class modification request (sending the system audio with a language label, generally 30 seconds or more per cut). A new language class will provide volatile scores unless sufficient cuts have been provided (approximately 10). In general, 30 minutes from around 30 samples is the minimum amount of data required to produce a reasonable language model. This enrollment can be augmented with subsequent class modification requests by adding more audio from the same language to an existing class, again, like SID or SDD. In addition to user enrolled languages, most LID plugins are supplied with several pre-enrolled languages. Users can replace these existing languages using their own data by enrolling audio with the same label as an existing language.
Note that LID enrollments are not independent. This means that the more languages the user enrolls from their domain, the better a given language they enroll will be.
Configuring Languages
Most LID plugins have the ability to re-configure the languages available in a domain. Configuring languages in the domain can be done by entering the domain directory of interest within the plugin folder and editing domain_config.txt. This file lists the pre-enrolled languages available in the plugin. Disabled languages are indicated by a # at the start of the line. To enable a language, remove the #. To disable a language, add a # at the start of the line.
Note that you cannot add languages to this list that are not supported by underlying models. If nonexistent language are added to this file, the plugin will intentionally fail.
Default Enabled Languages
The following languages are identified as high-confidence languages, supported by a sufficient amount of training data to make them reliable language detectors. As such, they are enabled by default in the plugin as-delivered, and serve as a general purpose base language set.
| Language Name | (cont'd) |
|---|---|
| Amharic | Modern Standard Arabic |
| English | Mandarin |
| French | Pashto |
| Iranian Persian | Portuguese |
| Japanese | Russian |
| Korean | Spanish |
| Levantine Arabic | Tagalog |
| Iraqi Arabic | Vietnamese |
Supported Languages
The full list of languages that exist as an enrolled class within this plugin as delivered are provided in the chart below. Note that as mentioned previously, not all of these languages were enrolled with enough data to serve as reliable detectors, but remain in the domain for the benefits to differentiating other languages, and for score calibration. If in doubt regarding whether an enrolled language should be used for detection or not, please reach out to SRI for clarification.
| Language Name | ||||
|---|---|---|---|---|
| Albanian | German | Modern Standard Arabic | Somali | Wu |
| Azerbaijani | Greek | Macedonian | Spanish | |
| Belarusian | Haitian | Mandarin | Swahili | |
| Bengali | Hausa | Min Nan | Tagalog | |
| Bosnian | Hindi | Ndebele | Tamil | |
| Bulgarian | Indonesian | Oromo | Thai | |
| Burmese | Iranian Persian | Panjabi | Tibetan | |
| Cantonese | Italian | Pashto | Tigrinya | |
| Cebuano | Japanese | Portuguese | Tunisian Arabic | |
| Croatian | Khmer | Romanian | Turkish | |
| English | Korean | Russian | Ukranian | |
| French | Levantine Arabic | Serbian | Uzbek | |
| Georgian | Iraqi Arabic | Shona | Vietnamese |
Global Options
This plugin offers several basic user-configurable parameters which can be edited directly in plugin_config.py or passed via the API. Note that if changed in the plugin_config.py file, a server running the plugin will need to be restarted in order to use the new parameters, while parameters passed via the API are dynamically updated and do not require a restart of the server.
The options available and their default values are described below:
| Option Name | Description | Default | Expected Range |
|---|---|---|---|
| min_speech | The minimum amount of detected speech in order to process a file. A higher value will prevent shorter files from being processed with the benefit of more reliable outputs. | 2.0 | 0.5 - 4.0 |
| sad_threshold | The threshold used to determine speech for processing. A higher value results in less speech detected, while removing more noise. | 1.0 | -3.0 - 6.0 |