sid-wav2vec-v2 (Speaker Identification and Diarization/Detection)

Version Changelog

Plugin Version	Change
v2.0.0	Initial plugin release. Released with OLIVE 6.2.0

Description

Speaker Identification (SID) plugins score a submitted segment of audio against one or more enrolled speakers with the goal of determining whether the speech in the segment in question was produced by one of the enrolled speakers. Speaker Detection (SDD) plugins go a step further, and allow the plugin to return timestamped and scored regions within the input audio where one or more known speakers are discovered. Speaker Diarization (DIA) plugins allow the former, timestamped regions, and are capable of differentiating and labeling unknown speakers.

This plugin is the first to combine all three, depending on how the plugin is tasked, by choosing which Trait is being exercised. The default behavior for each of this plugins' traits follows.

Global Scoring

Performs traditional Speaker ID, which assumes there is only one speaker present in the input audio, and provides scores representing the likelihood of if that speaker is one of the enrolled target speakers.

Region Scoring

First, performs blind speaker diarization, to determine how many different speakers are present, and label where each speaker is located. Then, if speaker enrollments are present, the plugin will use those identified "unknown speaker 1...N" regions, and score each against the target speaker enrollments to determine if any of the unknown speaker regions are likely to belong to one of the enrolled speakers. The output (example below) may contain only "unknownspkXX" regions, or it may contain only regions labeled with detections of enrolled speakers, or it may contain a mix of the two.

Low Level Details

The sid-wav2vec plugin uses a pre-trained 300 million parameter model as a speech data abstraction method prior to passing through a speaker-targeted time-delay neural network (TDNN). This TDNN was trained with data from approximately 14k speaker across multiple datasets. The wav2vec model was trimmed to take the output of the 12th layer rather than the 24th due to increased robustness and reduced computation needed in the similar research domain of speaker recognition. Embeddings extracted from the TDNN are passed through a Discriminate Condition Aware (DCA)-PLDA classifier. This classifier accounts for condition variation between trial sides, such as noise and compression level, and duration of content to produce well-calibrated log likelihood ratios even in unseen conditions.

In contrast to the previous sid-dplda-v3 plugin, this plugin replaces the input to the TDNN with an intermediate wav2vec abstraction rather than PNCC features, and an improved DCA-PLDA model accounting for more variation across our large test bed of data. Leveraging the wav2vec model provides more robustness across all conditions including distant speech, and allows speaker detection to be performed with less speech audio.

This plugin does both Speaker Identification (SID) as well as Speaker Detection (SDD). Speaker Detection plugins will detect and label regions of speech in a submitted audio segment where speech from one or more enrolled speakers are detected. This is in contrast to Speaker Identification (SID), which produces a score over the entire file for each enrolled speaker. So, unlike Speaker Identification, SDD is capable of handling audio where multiple speakers speak, and will provide timestamp region labels to point to the locations when speech from one or more of the enrolled speakers is found. Additionally, unknown unique speakers are output to indicate where speech originates from an un-enrolled speaker, or the strength of comparison is insufficient to label a known speaker.

Advancing on the previous method of speaker clustering in sdd-embed-v2, the sid-wav2vec-v2 plugin first extracts multiple embeddings from overlapping windows of speech that are only 1.0 second in duration as opposed to 4.0 seconds. This allows for improved region boundaries between different speakers. Linkage clustering is then used to gradually increase the number of speaker clusters until two or more clusters become too similar via DCA-PLDA scoring thus reverting back to the previous cluster count. This method is based on cluster-duration-specific thresholds which was found to reduce threshold sensitivity which is commonplace in diarization systems. This new approach to speaker clustering provided superior performance across 7 diarization datasets compared to the previous sdd-embed-v2 plugin.

Domains

• multi-v1
  • Multi-condition domain tested heavily on telephone and microphone conditions, multiple languages, distances, and varying background noises and codecs.

Inputs

For enrollment, an audio file or buffer with a corresponding speaker identifier/label. For scoring, an audio buffer or file.

Outputs

Generally, a list of scores, one for each of the speakers enrolled in the domain, for the entire segment. As with SAD and LID, scores are log-likelihood ratios where a score of greater than 0 is considered a detection. SID plugins, in particular, due to their association with forensics are generally calibrated or use dynamic calibration to ensure valid log-likelihood ratios to facilitate detection. Plugins may be altered to return only detections, rather than a list of enrollees and scores, but this is generally done on the client side for the sake of flexibility.

As noted above, this plugin now combines Global Scoring and Region Scoring; meaning it performs traditional Speaker ID (assumes the file is a single speaker, and returns scores for the entire file), or Speaker Detection (assume the speaker may change, and return scored and timestamped regions where speakers are detected within the file), depending on how it is called.

Global Score Output

Example output excerpt when performing traditional SID (Global Score):

/data/sid/audio/file1.wav speaker1 -0.5348
/data/sid/audio/file1.wav speaker2 3.2122
/data/sid/audio/file1.wav speaker3 -5.5340
/data/sid/audio/file2.wav speaker1 0.5333
/data/sid/audio/file2.wav speaker2 -4.9444
/data/sid/audio/file2.wav speaker3 -2.6564

Region Score Output

An example output excerpt when performing Speaker Diarization and Detection (SDD) (Region Score):

/data/sid/audio/file1.wav 8.320 13.110 unknownspk00 1.0000
/data/sid/audio/file1.wav 13.280 29.960 unknownspk01 1.0000
/data/sid/audio/file1.wav 30.350 32.030 unknownspk00 1.0000
/data/sid/audio/file2.wav 32.310 46.980 Phil 2.5333
/data/sid/audio/file2.wav 47.790 51.120 unknownspk02 1.0000
/data/sid/audio/file2.wav 54.340 55.400 unknownspk00 1.0000
/data/sid/audio/file2.wav 57.210 74.200 Mitch 9.5673

Note that in this example, some of the regions were determined to belong to a known/target speaker, and some were flagged as unknown, triggering that there were no target speaker detections in those regions.

Note also that when there is no detected speaker, a placeholder score of '1.0000' is provided. When a speaker is detected, the corresponding log-likelihood ratio score that region scored against the detected enrollment is provided.

Enrollments

SID plugins allow for class modifications. A class modification is essentially the capability to enroll a class with sample(s) of a class's speech - in this case, a new speaker. A new enrollment is created with the first class modification, which consists of essentially sending the system an audio sample from a speaker, generally 5 seconds or more, along with a label for that speaker. This enrollment can be augmented with subsequent class modification requests by adding more audio with the same speaker label.

Stateless Enrollments / Vectorization

This plugin now supports Stateless Enrollment. For more information on stateless enrollment, refer to the previous link.

Functionality (Traits)

The functions of this plugin are defined by its Traits and implemented API messages. A list of these Traits is below, along with the corresponding API messages for each. Click the message name below to go to additional implementation details below.

• GLOBAL_SCORER – Score all submitted audio, returning a single score for the entire audio segment for each of the enrolled speakers of interest.
  • GlobalScorerRequest
• REGION_SCORER – Score all submitted audio, returning labeled regions within the submitted audio, where each region includes a detected speaker of interest and corresponding score for this speaker.
  • RegionScorerRequest
• CLASS_MODIFIER – Enroll new speaker models or augment existing speaker models with additional data.
  • ClassModificationRequest
  • ClassRemovalRequest

Compatibility

OLIVE 6.2+

Limitations

Known or potential limitations of the plugin are outlined below.

Detection Granularity

All current SID plugins when performing a Global Score Request assume that an audio segment contains only a single speaker and may be scored as a single unit. If a given segment contains multiple speakers, the entire segment will still be scored as a unit. When the input is suspected to contain multiple speakers, or timetamped regions are needed as part of the output, then Region Scoring should be used, to perform Speaker Diarization and Detection (SDD).

Minimum Speech Duration

The system will only attempt to perform speaker identification if the submitted audio segment contains more than X seconds of detected speech (configurable as min_speech, 1.0 seconds by default).

Comments

GPU Support

This plugin was designed and developed to run optimally on GPU hardware. It is capable of running on CPU in the absence of an available GPU or the proper configuration, but it will do so at a significantly reduced speed.

Please refer to the OLIVE GPU Installation and Support documentation page for instructions on how to enable and configure GPU capability in supported plugins. By default this plugin will run on CPU only.

Minimum Speech

The plugin will only process files with at least 1.0 seconds of detected speech (configurable).

Global Options

This plugin offers several basic user-configurable parameters which can be edited directly in plugin_config.py or passed via the API. Note that if changed in the plugin_config.py file, a server running the plugin will need to be restarted in order to use the new parameters, while parameters passed via the API are dynamically updated and do not require a restart of the server.

These options are for expert users only. It is advisable to keep a backup of the defaults and to change with caution.

The options available and their default values are described below:

Option Name	Description	Default	Expected Range
threshold	Detection threshold: Higher value results in less detections being output, but of higher reliability.	0.0	-10.0 to 20.0
min_speech	The minimum length that a speech segment must contain in order to be scored/analyzed for the presence of enrolled speakers.	1.0	0.5 - 4.0
sad_threshold	SAD threshold for determining the audio to be used in metadata extraction	0.0	-5.0 - 6.0