lid-wav2vec-v1 (Language Identification and Diarization/Detection)
Version Changelog
| Plugin Version | Change |
|---|---|
| v1.0.0 | Initial release of plugin. Released with OLIVE 6.2.0. |
Description
LID plugins analyze an audio segment to produce a detection score for each of the enabled language or dialect classes for the domain in use. A plugin domain could consist of 50 or more languages and dialects in a single plugin, or as few as one for use cases where the customer is only focused on a single target class. Some plugin domains are solely focused on dialect or sub-language recognition, such as languages of China. Several LID plugins allow users to add new classes or augment existing classes with more data for the class to improve accuracy.
Low Level Details
The lid-wav2vec plugin uses a pre-trained 300 million parameter model as a speech data abstraction method prior to passing through a language-targeted time-delay neural network (TDNN). This TDNN was trained with approximately 250k utterances from 101 languages. The wav2vec model was trimmed to take the output of the 18th layer rather than the 24th due to increased robustness and reduced computation needed in the similar research domain of speaker recognition. Embeddings extracted from the TDNN are passed through a Gaussian Backend classifier trained to discriminate 101 languages, with 17 enabled by default. A linear calibration model is applied to the scores to convert them to log likelihood ratios.
In contrast to the previous lid-embed-v2 plugin, this plugin replaces the input to the TDNN with an intermediate wav2vec abstraction rather than ASR Bottleneck features. Leveraging the wav2vec model provides more robustness across all conditions including distant speech, and allows language detection to be performed with less speech audio. A future release of this wav2vec plugin will include the HDPLDA backend from our lid-hdplda-v2 plugin which we anticipate will provide added robustness to various conditions along with improved discriminate power.
Important Usage Notes
This plugin supports BOTH Global Scoring and Region Scoring; previously described as Language Identification (LID) and Language Detection (LDD).
This plugin does both Language Identification (LID) as well as Language Detection (LDD). Language Detection plugins will detect and label regions of speech in a submitted audio segment where one or more enrolled languages are detected being spoken. This is in contrast to Language Identification (LID) plugins, which label the entire segment with a single language. So, unlike Language Identification, LDD is capable of handling audio where multiple languages are being spoken, and will provide timestamp region labels to point to the locations when speech from one or more of the enrolled languages is found.
In contrast to prior LDD plugins, this plugin is capable of Language Diarization, allowing it to output regions of languages unknown to the plugin, or with insufficient information to produce a score above the detection threshold and receive a known language label. Diarization is performed by extracting embeddings for multiple overlapping windows of speech (4 seconds by default) and clustering them using linkage clustering via a configurable diarization threshold.
Domains
- multi-v1
- Generic domain for most close talking conditions with signal-to-noise ratio above 10 dB. Currently set up with 17 languages configured (optionally configurable to up to over 100 languages). See below for the currently-configured and available languages. See the configuring languages section for instructions on reconfiguring the available languages if necessary.
Inputs
Audio file or buffer and an optional identifier.
Outputs
Generally, a list of scores for all classes in the domain, for the entire segment. As with SAD and SID, scores are generally log-likelihood ratios where a score of greater than “0” is considered a detection. Plugins may be altered to return only detections, rather than a list of classes and scores, but this is generally done on the client side for sake of flexibility.
As noted above, this plugin now combines Global Scoring and Region Scoring; meaning it performs traditional Language ID (assumes the file is a single language, and returns scores for the entire file), or Language Detection (assume the language may change, and return scored and timestamped regions where languages are detected within the file), depending on how it is called.
Global Score Output
An example output excerpt when performing traditional LID (Global Score):
input-audio.wav Amharic -6.73527861
input-audio.wav Arabic -3.31796265
input-audio.wav English 8.22701168
input-audio.wav French -2.98071671
input-audio.wav Iranian Persian -5.55558729
input-audio.wav Japanese -6.01283073
input-audio.wav Korean -5.64162636
input-audio.wav Mandarin -4.81163836
input-audio.wav Portuguese -1.93523705
input-audio.wav Russian -5.60199690
input-audio.wav Spanish -3.70800495
input-audio.wav Tagalog -4.86510944
input-audio.wav Vietnamese -5.10995102
Region Score Output
LDD plugins return a list of regions with a score for each detected language. The starting and stopping boundaries are denoted in seconds. As with LID above, scores are log-likelihood ratios, where a score greater than the default threshold of "0" is considered to be a detection.
An example output excerpt when performing LDD (Region Score):
input-audio.wav 0.000 41.500 French 11.40590000
input-audio.wav 43.500 77.500 French 11.29558277
input-audio.wav 78.500 80.500 Levantine Arabic 2.25519705
input-audio.wav 85.500 86.500 French 2.06612849
input-audio.wav 97.500 98.500 French 3.74665093
input-audio.wav 98.500 99.500 Mandarin 2.22936487
input-audio.wav 105.500 106.500 Spanish 2.72254372
input-audio.wav 107.500 108.500 French 2.60355234
input-audio.wav 108.500 110.500 French 2.76414633
input-audio.wav 109.500 113.140 English 2.85003138
input-audio.wav 113.760 116.260 French 2.50716114
input-audio.wav 120.260 140.260 Korean 14.93032360
input-audio.wav 143.260 157.260 Korean 12.62243176
input-audio.wav 158.260 161.260 Mandarin 3.24917603
input-audio.wav 161.260 162.260 French 2.73345900
input-audio.wav 165.260 177.260 Korean 12.32051945
input-audio.wav 178.260 180.260 Levantine Arabic 2.41706276
input-audio.wav 186.320 188.820 Levantine Arabic 2.85040617
input-audio.wav 193.820 194.820 Spanish 2.21501803
Functionality (Traits)
The functions of this plugin are defined by its Traits and implemented API messages. A list of these Traits is below, along with the corresponding API messages for each. Click the message name below to go to additional implementation details below.
- GLOBAL_SCORER – Score all submitted audio, returning a single score for the entire audio segment for each of the enrolled and enabled languages of interest.
- REGION_SCORER – Score all submitted audio, returning labeled regions within the submitted audio where each region includes a detected language of interest and corresponding score for this language
- CLASS_MODIFIER – Enroll new language models or augment existing language models with additional data.
Compatibility
OLIVE 6.2+
Limitations
Known or potential limitations of the plugin are outlined below.
All current LID plugins when performing a Global Score assume that an audio segment contains only a single language and may be scored as a unit. If a segment contains multiple languages the entire segment will still be scored as a unit. In many cases, a minimum duration of speech of 1 second(s) is required in order to output scores. This value can optionally be overwritten, but scores provided for such short segments will be volatile.
Minimum Speech Duration
The system will only attempt to perform language identification if the submitted audio segment contains more than 1 second(s) of detected speech.
Languages of Low Confidence
Many of the language models that are included and hidden within the domain's data model, disabled by default, do not contain enough data within the model for reliable detection of this language, and are included solely to help with score calibration, and differentiating other languages. If in doubt regarding whether an enrolled language should be used for detection or not, please reach out to SRI for clarification.
Comments
GPU Support
This plugin was designed and developed to run optimally on GPU hardware. It is capable of running on CPU in the absence of an available GPU or the proper configuration, but it will do so at a significantly reduced speed.
Please refer to the OLIVE GPU Installation and Support documentation page for instructions on how to enable and configure GPU capability in supported plugins. By default this plugin will run on CPU only.
Language/Dialect Detection Granularity
LID plugins attempt to distinguish dialects (ie., Tunisian Arabic and Levantine Arabic) or a base language class (such as Arabic). These can be mapped back to the base language if desired. This requires one change to be enabled.
- A mapping file 'dialect_language.map' must exist within the domain of the plugin for which mapping is to be performed (eg. domains/multi-v1/dialect_language.map). This file is a tab-delimited, two-column file that lists each mapping for the dialect to the languages as "
\t ". Example lines include: Levantine Arabic Arabic Tunisian Arabic Arabic Iranian Persian Farsi
In the example above, the output labels of the dialects will be mapped to the same base language 'Arabic'. Note the exception in which mapping is not performed is for user-enrolled languages where it is assumed the user has provided the dialect or language label based on their requirements.
Note that we recommend users request these mapping files from SRI, or request the mapping to be performed before delivery of the plugin so that SRI can test and validate the final mapping before delivery.
Note also that the system will not allow you to create an enrollment with the same class name that you have languages mapped to. This is to avoid confusing situations where the system isn't sure if it should be considering the original pre-mapped models, or the newly enrolled user model. You must provide a unique name for any new language enrollments, that does not conflict with the dialect_langage.map. If you have already enrolled a conflicting model, and then add a mapping to this same name, the plugin will provide a warning message and intentionally fail to load.
Enrollments
Some recent LID plugins allow class modifications. A class modification is essentially an enrollment capability similar to SID. A new enrollment is created with the first class modification request (sending the system audio with a language label, generally 30 seconds or more per cut). A new language class will provide volatile scores unless sufficient cuts have been provided (approximately 10). In general, 30 minutes from around 30 samples is the minimum amount of data required to produce a reasonable language model. This enrollment can be augmented with subsequent class modification requests by adding more audio from the same language to an existing class, again, like SID or SDD. In addition to user enrolled languages, most LID plugins are supplied with several pre-enrolled languages. Users can replace these existing languages using their own data by enrolling audio with the same label as an existing language.
Note that LID enrollments are not independent. This means that the more languages the user enrolls from their domain, the better a given language they enroll will be.
Configuring Languages
Most LID plugins have the ability to re-configure the languages available in a domain. Configuring languages in the domain can be done by entering the domain directory of interest within the plugin folder and editing domain_config.txt. This file lists the pre-enrolled languages available in the plugin. Disabled languages are indicated by a # at the start of the line. To enable a language, remove the #. To disable a language, add a # at the start of the line.
Note that you cannot add languages to this list that are not supported by underlying models. If nonexistent language are added to this file, the plugin will intentionally fail.
Note that internally, this plugin uses ISO-639-3 Language Codes to refer to each language. They are translated to English language names before being reported by OLIVE for human consumption, but it's important to know the language code when enabling or disabling a language. Refer to the link above to look up language codes, or see below for a list of the included languages and a mapping of the internal codes to the reported language name.
Default Enabled Languages
The following languages are identified as high-confidence languages, supported by a sufficient amount of training data to make them reliable language detectors. As such, they are enabled by default in the plugin as-delivered, and serve as a general purpose base language set.
| Language | Default Mapped/Reported Language Name |
|---|---|
| Amharic | Amharic |
| English | English |
| French | French |
| Indonesian | Indonesian |
| Iranian Persian | Farsi |
| Japanese | Japanese |
| Khmer | Khmer |
| Korean | Korean |
| Levantine Arabic | Arabic |
| Mandarin | Mandarin |
| Pashto | Pashto |
| Portuguese | Portuguese |
| Russian | Russian |
| Spanish | Spanish |
| Ukrainian | Ukrainian |
| Vietnamese | Vietnamese |
Supported Languages
The full list of languages that exist as an enrolled class within this plugin as delivered are provided in the chart below. Note that as mentioned previously, not all of these languages were enrolled with enough data to serve as reliable detectors, but remain in the domain for the benefits to differentiating other languages, and for score calibration. If in doubt regarding whether an enrolled language should be used for detection or not, please reach out to SRI for clarification.
| Language | |||
|---|---|---|---|
| Abkhazian | Afrikaans | Albanian | Amharic |
| Armenian | Assamese | Azerbaijani | Bashkir |
| Basque | Belarusian | Bengali | Breton |
| Bulgarian | Burmese | Cantonese | Catalan |
| Cebuano | Czech | Danish | Dutch |
| Egyptian Arabic | English | Estonian | Faroese |
| Finnish | French | Galician | Georgian |
| German | Greek | Gujarati | Gulf Arabic |
| Haitian | Hausa | Hebrew | Hungarian |
| Icelandic | Indonesian | Iranian Persian | Iraqi Arabic |
| Italian | Japanese | Javanese | Kannada |
| Kazakh | Khmer | Korean | Latvian |
| Levantine Arabic | Lingala | Lithuanian | Luxembourgish |
| Macedonian | Malagasy | Malayalam | Maltese |
| Mandarin | Maori | Marathi | Min Nan |
| Mongolian | Ndebele | Nepali | Norwegian Nynorsk |
| Occitan | Oromo | Panjabi | Pashto |
| Polish | Portuguese | Romanian | Russian |
| Serbo-Croatian | Shona | Sindhi | Sinhala |
| Slovak | Slovenian | Somali | Spanish |
| Sundanese | Swahili | Swedish | Tagalog |
| Tajik | Tamil | Tatar | Telugu |
| Thai-Lao | Tibetan | Tigrinya | Tunisian Arabic |
| Turkish | Turkmen | Ukrainian | Urdu-Hindi |
| Uzbek | Vietnamese | Wu | Yiddish |
| Yoruba |
Global Options
This plugin offers several basic user-configurable parameters which can be edited directly in plugin_config.py or passed via the API. Note that if changed in the plugin_config.py file, a server running the plugin will need to be restarted in order to use the new parameters, while parameters passed via the API are dynamically updated and do not require a restart of the server.
These options are for expert users only. It is advisable to keep a backup of the defaults and to change with caution.
The options available and their default values are described below:
| Option Name | Description | Default | Expected Range |
|---|---|---|---|
| threshold | The minimum score for a language to be considered a 'detection.' Lower values will cause the plugin to be more permissive, and may result in more false alarms. | 2.0 | 0.5 - 4.0 |
| max_class_per_file | If set to a number larger 0, will only output the top scoring max_class_per_file classes. This is a way to limit the output. | 0 | - |
| max_class_per_frame | For Region Scoring, default True, only outputs a single class per region when using LDD (Region Scoring) | True | - |
| enable_diarization | Allows clustering of same languages prior to running detection. If False, runs detection on small windows (default 4.0 seconds) and merges results. Enabling diarization is considered more accurate. | True | - |
| enable_diarization_unknown_lang_output | If enable_diarization == True, then for a language cluster that is not detected as a known language, a label ‘unknownlanguage00’ will be output for the region, incrementing for each new language cluster (different from the prior cluster) that is detected. The label may occur at different parts of the file, still grouping that same language under one label. To clarify, a single audio file may have multiple unknownlanguage00 regions throughout a file with known (such as ‘English’) or other unknown language labels (such as ‘unknownlanguage01’). | True | - |
| dia_threshold | Fine tuning for clustering algorithm to decide how many unique language clusters are in a file. Lower creates more clusters. | 0.10 | - |
| dia_max_clusters | The maximum number of language clusters that can be found. | 7 | 2 - 12 |