Integrating a Client API with OLIVE
Java Client IDE Setup Guide
If you would like to import the provided OLIVE Java API sample code and reference implementation into an IDE to explore the code and/or get started with integrating this code, please refer over to the Java Client IDE Setup Guide page.
If you've already done this step, plan on integrating this functionality just using the provided JAR file, or just wish to browse code samples, continue below.
The OLIVE Java API
While the OLIVE system allows client integration via Protobuf messages sent from a variety of languages such as Java, Python, C++ and C#, and machine types such as Windows, Linux, and MacOS, the currently available OLIVE reference API implementation is Java-based only, and is also referred to as the OLIVE Java Client API. Therefore all instructions and code examples presented in this section currently assume that client programs are also in Java. Pseudocode and Python versions of the included code will be rolled into this page as they are available.
Fundamentally any OLIVE API implementation is based on exchanging Protobuf messages between the client and server. These messages are defined in the API Message Reference Documentation. If you would like to create a reference implementation of the OLIVE API in another language, please refer to the information in the Developing an OLIVE API Reference Implementation page, that should prove to be helpful. If you would like an API reference implementation in another language to be created, please reach out to olive-support@sri.com to discuss your needs with the team.
Basic Recipe for using the OLIVE Java Client API
This section covers the first steps towards building a client based on implementing SRI's OLIVE API Java Reference implementation in order to use the OLIVE Java API for speech processing. The sections immediately following cover setting up an OLIVE server instance to connect to, then move into a number of steps necessary to connect a client to this server, and finally cover how to build and submit scoring and enrollment requests from the client program.
The first step towards integrating a client with the provided OLIVE Java Reference Implementation is to review the API Message Reference to understand the available request and response messages, then dive in by setting up your IDE to work with this code. We have created a step-by-step guide for this configuration and setup process. Note that these steps are not necessary for implementing the actual functionality from this software package, but could be very useful for exploring and learning the code. If only the functionality from this reference implementation is desired, SRI can provide an appropriate JAR file to include in your project.
OLIVE server
As mentioned above, the OLIVE Enterprise API operates on a client/server model, making the OLIVE server an essential part of the OLIVE system. You must run the OLIVE server and manage its lifecycle as part of your integration effort. For more information about the server's duties, how to interact with it, what options and functionality are available to it, and general setup and troubleshooting information, please refer to the OLIVE Server Information Page.
Establish Server Connection
Before making any task request, a client must establish a connection with the server. By default, the OLIVE server listens on ports 5588 (request port) and 5589 (status port) for client connection and status requests. These ports are configurable, but if the server has not been instructed to change its listening ports, the code below should establish a connection.
Using the OLIVE Java API, connection to the server can be made with a single call, as shown below. This call is available from the API Server class, included in the package com.sri.scenic.api.Server.
Server server = new Server();
server.connect(
"scenic-client", //client-id
"localhost", //address of server
5588, //request-port
5589, //status-port
10000 //timeout for failed connection request
);
--- Coming Soon ---
connect(server-host-name, 5588, 5589)
The request port (5588 by default) is used for call and response messages (Protobuf messages). Each request message sent to this port is guaranteed a response from the server (this is why the messages in the API Message Reference are often suffixed with 'Request' and 'Result'). There is no need to poll the server for information on a submitted request, as the result/response for the a request is returned to the client as soon as it is available.
The status port (5589 by default) is used by the Server to publish health and status messages (Heartbeat) to client(s). Clients can not send requests on this port.
Request Available Plugins
In order to submit most server requests, the client must specify the plugin and domain to handle the request. To obtain the handle of a targeted plugin, the client first requests a list of all currently available valid plugins from the server. From the returned plugins list, the client looks up the specific plugin handle by the plugin's unique name (id) and its associated trait for the task to be performed. This handle can then be used in a future request message sent to the server.
In the Java API Reference Implementation, this is accomplished by the utilities requestPlugin() and findPluginDomainByTrait(), within the included ClientUtils package. The steps are shown below.
// ask the server for a list of currently available plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// Look up a specific plugin from the plugin list using
// the plugin's unique name (pluginName and domainName) and associated trait e.g. FRAME_SCORER
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, "SAD", Scenic.TraitType.FRAME_SCORER, pluginList);
def request_plugins(self):
request = PluginDirectoryRequest()
# Wrap message in an Envelop
request = self._wrap_message(request, FRAME_SCORER_REQUEST)
# Now send the message
logging.debug("Sending a Plugin request message")
self.request_socket.send(request.SerializeToString())
# Wait for the response from the server
# logging.info("checking for response")
protobuf_data = self.request_socket.recv()
logging.info("Received message...")
envelope = Envelope()
envelope.ParseFromString(protobuf_data)
# for this use case the server will only have one response in the envelope:
for i in range(len(envelope.message)):
server_msg = envelope.message[i]
if broker_msg.HasField("error"):
print(server_msg.error)
else:
plugin_dir_msg = PluginDirectoryResult()
plugin_dir_msg.ParseFromString(server_msg.message_data[0])
# There is likely to be multiple plugin/domains
print("Server has {} plugins ".format(len(plugin_dir_msg.plugins)))
for p in plugin_dir_msg.plugins:
print("Plugin {} has {} domains".format(p.id, len(p.domain)))
plugin-list = getListOfAvailablePlugins(server);
targeted-plugin = lookupPlugin(plugin-list,pluginName,pluginTrait);
The targeted plugin handle, pd in the above example, can then be used with other utilities within the Client API to submit requests, or to otherwise interact with the plugin. For example, this code below shows how one might check what Traits that each plugin in the pluginList returned above support:
for(Pair<Scenic.Plugin, Scenic.Domain> pair : pluginList){
for(Scenic.Trait t : pair.getFirst().getTraitList()) {
if(t.getType() == Scenic.TraitType.FRAME_SCORER){
log.info("Plugin {} supports frame scoring", pair.getFirst().getId());
}
if(t.getType() == Scenic.TraitType.GLOBAL_SCORER){
// Supports global scoring (i.e. SID or LID)
log.info("Plugin {} supports global scoring", pair.getFirst().getId());
}
if(t.getType() == Scenic.TraitType.REGION_SCORER){
// Supports region scoring (i.e. KWS)
log.info("Plugin {} supports region scoring", pair.getFirst().getId());
}
}
}
--coming soon--
--coming soon--
Audio Submission Guidelines
One of the core client activities is submitting Audio with a request. In the OLIVE API, three ways are provided for a client to package audio data to send to the OLIVE server:
- file path
- buffer of raw audio sample data
- serialized file buffer object
The enum AudioTransferType is used to specify they type of audio transfer to use.
When the client and the OLIVE server share the same file system, the easiest way for the client to send audio data to the server is by specifying the audio's file path on disk. The OLIVE Client API provides the utility below to package audio files which are accessible to the server locally:
packageAudioAsPath() // AudioTransferType.SEND_AS_PATH
When the client and the server don't share the same file system, as in the case of a client making a remote connection to the OLIVE server, it is necessary to send the client's local audio files as a file buffer. To help package the client's audio data in a raw buffer, the OLIVE Client API provides the utility below:
packageAudioAsRawBuffer() // AudioTransferType.SEND_SAMPLES_BUFFER
When submitting audio to the server as a buffer of raw samples, it is important to include information characterizing the audio, such as the bit depth, audio encoding, sample rate, and number of channels, to ensure the server knows how to properly treat the buffer it receives.
A third utility, shown below, also packages the client's audio data in a buffer. This utility passes the original file to the server in its entirety in one contiguous buffer, leaving the audio file header intact.
packageAudioAsSerializedBuffer() // AudioTransferType.SEND_SERIALIZED_BUFFER
Sending audio data as a serialized file buffer ensures that all audio header information is provided intact to the server. This allows the server to properly decode and process the audio once its received, since it can directly access the bit depth, encoding type, sample rate and other necessary information from the header itself. The tradeoff with serialized files is that there may be additional overhead needed to process the audio into a consumable form. If the client and server reside on the same hardware and file system, it is advisable to simply pass filepaths when possible. This saves the memory overhead burden of both the client and server loading audio into memory. If using common audio types, like 16-bit PCM .wav files, it may also be possible to simply pass a non-serialized file buffer.
The OLIVE Java Client API provides utilities such as requestFrameScore(), requestEnrollClass(), etc. to handle various client message requests that were covered above. In these utilities, the enum argument transferType is used to select in what way the audio data is to be sent. For example, when transferType is set to AudioTransferType.SEND_SERIALIZED_BUFFER, audio data will be interpreted as if it were sent as a serialized buffer.
Synchronous vs. Asynchronous Message Submission
The OLIVE Client API allows the client to choose between processing a task request synchronously or asynchronously. Processing a task request synchronously means the client will block and wait for the task result to return before proceeding to other task requests. On the other hand, asynchronous processing means the client will not wait for the result to come back before moving on, allowing several jobs to be submitted in parallel. The examples below generally show submitting requests asynchronously.
The argument async in the API utilities requestFrameRequest(), requestEnrollClass(), etc., can be used to select if the client intends to wait for the task result to return. When async is set to true, the client will not block when a request is sent to the server, so other task requests can be made before the results are received asynchronously and handled by the callback.
Construct Request Message
Information contained in a task request message may be different depending on the type of task to be performed. For a scoring task, the request message usually contains specific plugin and domain names, and the audio to be scored. For an adaptation or enrollment task, it also contains class ID information.
To make a task request, the client program must first assemble the necessary information into a request message, and then send the message to the server. These 2 steps are shown in pseudo code below.
request=packageRequest(pluginName,pluginDomain,requestType, any other information specific to requestType)
sendRequest(server,request,audio)
Client API Code Samples
The OLIVE Reference API includes functionality pre-coded to accommodate many of the available request messages. Utilities such as requestFrameScore(), requestEnrollClass(), etc. not only do the packaging of request messages, but also take care of sending the request messages to the server, all in one call. They are available from the SRI Java API package sri.com.scenic.api.ClientUtils.
The required parameters to send requests using these scoring utilities include:
- server handle (server - see here)
- the plugin handle (pd - see here)
- the name of the audio file to submit to the server (filename)
- channel number of the audio to be processed when audio has more than 1 channel (channelNumber)
- a callback function for handling results returned either asynchronously or synchronously (rc)
- whether the client will block for task result to return (async - see here)
- an enum of how to submit audio to the server (transferType)
- optional lists of annotations of the submitted audio (regions)
- optional list of parameters for customizing plugin behavior (options)
- optional list of class IDs for filtering the results (classIDs)
Performing an enrollment request adds an additional parameter:
- the ID of the class to be enrolled
The primary requests covered below are:
- Frame score requests - used to make most SAD requests (note that some SAD plugins return regions, not frame scores)
- Global score requests - used to make LID, SID, or Gender score requests
- Region score requests - used to make QbE, KWS, Diarization, and sometimes SAD score requests
- Enrollment requests - used to enroll speakers or other class types for plugins that support the ClassEnroller trait
Frame Score Request
The example below provides sample code for a function MyFrameScoreRequest that takes a server connection, a plugin/domain handle, and a path to an audio file as arguments, and uses this information to build and submit a Frame scoring request to the connected server.
Note that this code passes the audio to the server using a simple file path, assuming that the client and server have a shared file system. It is also possible to perform this request using buffered audio samples or a serialized file.
Note that only a plugin that support the FrameScorer trait, cna handle this request. All SAD plugins support this train, while some also support the RegionScorer trait.
For an example of how to call this code with a specific plugin, refer to the SAD Scoring Request code example below.
The full Java code file this example was pulled from is also available, showing the process of establishing a server connection, and polling the server for available plugins to retrieve the appropriate plugin/domain handle.
public static boolean MyFrameScoreRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String filename) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// Create a callback to handle SAD results from the server
Server.ResultCallback<Scenic.FrameScorerRequest, Scenic.FrameScorerResult> rc = new Server.ResultCallback<Scenic.FrameScorerRequest, Scenic.FrameScorerResult>() {
@Override
public void call(Server.Result<Scenic.FrameScorerRequest, Scenic.FrameScorerResult> r) {
// output frame scores
if (!r.hasError()) {
for (Scenic.FrameScores fs : r.getRep().getResultList()) {
System.out.println(String.format("Received %d frame scores for '%s'", fs.getScoreCount(), fs.getClassId()));
Double[] scores = fs.getScoreList().toArray(new Double[fs.getScoreList().size()]);
int rate = fs.getFrameRate();
for (int i = 0; i < scores.length; i++)
{
int start = (int) (100 * i / (double) rate);
int end = (int) (100 *(i+1) / (double) rate);
System.out.println(String.format("start: '%d' end: '%d' score:'%f'",start,end,scores[i]));
}
}
}
System.exit(0);
}
};
return ClientUtils.requestFrameScore(server, pp, filename, 1, rc, true, true, regionParser.getRegions(filename), new ArrayList<>(), new ArrayList<>());
}
---Coming Soon---
--Coming Soon--
Global Score Request
The example below provides sample code for a function MyGlobalScorerRequest that takes a server connection, a plugin/domain handle, and a path to an audio file as arguments, and uses this information to build and submit a Global scoring request to the connected server.
Note that this code passes the audio to the server using a simple file path, assuming that the client and server have a shared file system. It is also possible to perform this request using buffered audio samples or a serialized file.
Note also that the code required to submit a GlobalScorerRequest message doesn't care what type of plugin is going to be doing the scoring, as long as the plugin implements the GlobalScorer Trait. This means that the exact same code can be used for submitting audio to global scoring SID plugins, LID plugins, or any other global scoring plugin.
For an example of how to call this code with a specific plugin, refer to the SID Scoring Request code example below.
The full Java code file this example was pulled from is also available, showing the process of establishing a server connection, and polling the server for available plugins to retrieve the appropriate plugin/domain handle.
public static boolean MyGlobalScorerRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String scoreWaveFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a call back to handle the global scoring request
Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> scoreCallback = new Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult>() {
@Override
public void call(Server.Result<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> r) {
// do something with the results:
if(!r.hasError()){
log.info("Received {} scores:", r.getRep().getScoreCount());
for(Scenic.GlobalScore gs : r.getRep().getScoreList()){
System.out.println(String.format("speaker{%s} = {%f}",gs.getClassId(),gs.getScore()));
}
}
else{
System.out.println(String.format("Global scorer error: {%s}", r.getError()));
}
System.exit(0);
}
};
// make the global scoring reqeust
return ClientUtils.requestGlobalScore(server, pp, Scenic.TraitType.GLOBAL_SCORER, scoreWaveFileName, 0, scoreCallback, true, true, regionParser.getRegions(scoreWaveFileName), new ArrayList<>(), new ArrayList<>());
}
def request_lid(self, plugin, domain, filename, classes):
'''
Request a LID analysis of 'filename'
:param plugin: the name of the LID plugin
:param domain: the name of the plugin domain
:param filename: the name of the audio file to score
:param languages: optional list of languages trigraphs used to filter (restrict) results returned from the server. This list should only include languages supported by the specified plugin
:return: the LID analysis as a list of (global) scores
'''
request = GlobalScorerRequest()
request.plugin = plugin
request.domain = domain
audio = request.audio
# send the name of the file to the server:
audio.path = filename
# alternatively, you could send an audio buffer:
# from scipy.io import wavfile
# sample_rate, data = wavfile.read(filename)
# package_buffer_audio(audio, data, data.shape[0], sample_rate)
request = self._wrap_message(request, GLOBAL_SCORER_REQUEST)
# Now send the message
logging.debug("Sending a LID (global score request) message")
self.request_socket.send(request.SerializeToString())
# Wait for the response from the server
# logging.info("checking for response")
protobuf_data = self.request_socket.recv()
logging.info("Received message from server...")
#Unpack message
envelope = Envelope()
envelope.ParseFromString(protobuf_data)
# for this use case the server will only have one response in the envelope:
for i in range(len(envelope.message)):
broker_msg = envelope.message[i]
if broker_msg.HasField("error"):
print broker_msg.error
else:
global_score_msg = GlobalScorerResult()
global_score_msg.ParseFromString(broker_msg.message_data[0])
# Assume there is only one result set (for 'speech'): frame_score_msg.result[0]
print("Received {} global scores".format(len(global_score_msg.result[0].score)))
return global_score_msg.result[0]
return None
---Coming Soon---
Region Score Request
The example below provides sample code for a function MyRegionScorerRequest that takes a server connection, a plugin/domain handle, and a path to an audio file as arguments, and uses this information to build and submit a Region scoring request to the connected server.
Note that this code passes the audio to the server using a simple file path, assuming that the client and server have a shared file system. It is also possible to perform this request using buffered audio samples or a serialized file.
For an example of how to call this code with a specific plugin, refer to the KWS Scoring Request code example below.
The full Java code file this example was pulled from is also available, showing the process of establishing a server connection, and polling the server for available plugins to retrieve the appropriate plugin/domain handle.
public static boolean MyRegionScorerRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String scoreWaveFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a call back to handle the global scoring request
Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> scoreCallback = new Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult>() {
@Override
public void call(Server.Result<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> r) {
// scoring result received
if(!r.hasError()){
log.info("Received {} region scores:", r.getRep().getRegionCount());
for (Scenic.RegionScore rs : r.getRep().getRegionList()) {
log.info("\t{} = {}. From {} to {} ", rs.getClassId(), rs.getScore(), rs.getStartT(), rs.getEndT());
}
} else {
log.error("SDD Region scoring error: {}", r.getError());
}
System.exit(0);
}
};
// make the region scoring reqeust
return ClientUtils.requestRegionScores(server, pp, scoreWaveFileName, 0, scoreCallback, true, false, regionParser.getRegions(scoreWaveFileName), new ArrayList<>(), new ArrayList<>());
}
---Coming soon---
--Coming soon--
Enrollment Request
The example below provides sample code for a function MyEnrollmentRequest that takes a server connection, a plugin/domain handle, the name of the class (speaker) to enroll, and a path to an audio file as arguments, and uses this information to build and submit a Region Scoring request to the connected server.
Examples below show an enrollment request with one file, followed by an enrollment with multiple files (for one class).
Enrollment Request with a single audio file/enrollment.
public static boolean MyEnrollmentRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String classID, String enrollmentFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a callback that handles the enrollment result from the server
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> enrollmentCallback = new Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult>() {
@Override
public void call(Server.Result<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> r) {
// examine enrollment result
if(!r.hasError()){
System.out.println("Enrollment succeeded");
}
else {
log.error("Enrollment request failed: {}", r.getError());
}
}
};
// do enrollment request
// make it synchronous so we know enrollment is complete
boolean enrolled = false;
enrolled = ClientUtils.requestEnrollClass(server, pp, classID, enrollmentFileName, 0, enrollmentCallback, false, false, new ArrayList<>(), enrollmentOptions);
/** can do something else now, enrollment is complete **/
return true;
}
---Coming soon---
---Coming soon---
Batch enrollment request with multiple files.
public static boolean MyEnrollmentUsingMultipleFilesRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String classID, ArrayList<String> audioFiles, int enrollmentFileCount) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a callback that handles the enrollment result from the server
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> enrollmentCallback = new Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult>() {
@Override
public void call(Server.Result<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> r) {
// do something with the results:
if(!r.hasError()){
System.err.println("*****server processed 1 enrollment file*****");
responseCount++;
}
else {
log.error("Enrollment request failed: {}", r.getError());
}
}
};
for (String audioFilename : audioFiles) {
// For simplicity, enrollments requests are done synchronously
boolean enrolled = ClientUtils.requestEnrollClass(server, pp, classID, audioFilename, 0, enrollmentCallback, false, AudioTransferType.SEND_AS_FILE, new ArrayList<>(), enrollmentOptions);
if (enrolled) {
// can do something here like print out enrollment file processed
}
}
// wait for all enrollment files to be processed
while(responseCount != enrollmentFileCount) {
try{
TimeUnit.SECONDS.sleep(10);
} catch (InterruptedException e) {
System.out.println("*****Timeout Exception*****");
}
}
return true;
// all enrollment files are processed, now we can proceed to do other things if necessary
}
---Coming soon---
---Coming soon---
The function call definitions for some of these scoring and enrollment request utilities follows:
boolean requestFrameScore(Server server,
Pair<Scenic.Plugin, Scenic.Domain> pp,
String filename,
int channelNumber,
Server.ResultCallback<Scenic.FrameScorerRequest, Scenic.FrameScorerResult> rc,
boolean async,
AudioTransferType transferType,
List<RegionWord> regions,
List<Pair<String, String>> options,
List<String> classIDs)
This call submits a frame scoring request to the OLIVE server, e.g. SAD scoring. The returned results are a list of regions (regions and scores) and possibly a list of associated class IDs (classIDs).
boolean requestGlobalScore(Server server,
Pair<Scenic.Plugin, Scenic.Domain> plugin,
Scenic.TraitType trait,
String filename,
int channelNumber,
Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> rc,
boolean async,
AudioTransferType transferType,
List<RegionWord> regions,
List<Pair<String, String>> options,
List<String> classIDs)
This call submits a global scoring request to the OLIVE server, e.g. LID scoring. The returned results are a list of regions (regions and scores) and possibly a list of associated class IDs (classIDs).
boolean requestRegionScores(Server server,
Pair<Scenic.Plugin, Scenic.Domain> plugin,
String filename,
int channelNumber,
Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> rc,
boolean async,
AudioTransferType transferType,
List<RegionWord> regions,
List<Pair<String, String>> options,
List<String> classIDs)
This call submits a region scoring request to the OLIVE server, e.g. KWS scoring. The returned results are a list of regions (regions and scores) and possibly a list of associated class IDs (classIDs).
boolean requestEnrollClass(Server server,
Pair<Scenic.Plugin, Scenic.Domain> pp,
String id,
String wavePath,
int channelNumber,
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> rc,
boolean async,
AudioTransferType transferType,
List<RegionWord> regions,
List<Pair<String, String>> options)
This call submits an enrollment request to the OLIVE server, e.g. enrolling speakers in a SID plugin. A class ID (id) is required for enrollment. If selected regions of an audio are used for enrollment, they are passed along in a list (regions).
To see examples of how these utilities are used, consult sample client code provided in the next section.
Plugin Specific Code Examples
This section shows examples of using the functions just outlined to make calls to specific plugins, and demonstrate how the same code can be reused for several purposes - for example, requestGlobalScore is valid to request scoring from both SID and LID plugins.
All client examples below can be found in the OLIVE-API-examples tree in the src/main/java/com/sri/scenic/api/client folder.
- SAD Scoring Example
- SID Enrollment and Scoring Example
- LID Enrollment and Scoring Example (synchronized approach)
- LID Enrollment and Scoring Example (asynchronized approach)
- KWS Scoring Example
- SDD Enrollment and Scoring Example
- SAD Supervised Adaptation Example
- TPD Enrollment and Scoring Example
SAD Scoring Request
This shows a full implementation of a client program which sends a frame scoring request to a SAD plugin. Upon return of the result, it outputs the received frame scores. Included also is a second version where a threshold is used to filter out frame scores which are higher.
package com.sri.scenic.api.client;
import com.google.protobuf.InvalidProtocolBufferException;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.*;
import com.sri.scenic.api.client.*;
import org.apache.commons.cli.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.UnsupportedAudioFileException;
import java.io.*;
import java.nio.file.*;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
public class MySADFrameScorer {
private static Logger log = LoggerFactory.getLogger(MySADFrameScorer.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "tel-v1";
private static String pluginName = "sad-dnn-v6_rc5";
private static RegionParser regionParser = new RegionParser();
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// audio file name is passed as an argument
String audioFileName = args[0];
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// from the list of plugins, find the targeted plugin for the task
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, "SAD", Scenic.TraitType.FRAME_SCORER, pluginList);
// formulate SAD frame scoring request and send to server
MyFrameScoreRequest(server, pd, audioFileName);
}
public static boolean MyFrameScoreRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String filename) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// Create a callback to handle SAD results from the server
Server.ResultCallback<Scenic.FrameScorerRequest, Scenic.FrameScorerResult> rc = new Server.ResultCallback<Scenic.FrameScorerRequest, Scenic.FrameScorerResult>() {
@Override
public void call(Server.Result<Scenic.FrameScorerRequest, Scenic.FrameScorerResult> r) {
// output frame scores
if (!r.hasError()) {
for (Scenic.FrameScores fs : r.getRep().getResultList()) {
System.out.println(String.format("Received %d frame scores for '%s'", fs.getScoreCount(), fs.getClassId()));
Double[] scores = fs.getScoreList().toArray(new Double[fs.getScoreList().size()]);
int rate = fs.getFrameRate();
for (int i = 0; i < scores.length; i++)
{
int start = (int) (100 * i / (double) rate);
int end = (int) (100 *(i+1) / (double) rate);
System.out.println(String.format("start: '%d' end: '%d' score:'%f'",start,end,scores[i]));
}
}
}
System.exit(0);
}
};
return ClientUtils.requestFrameScore(server, pp, filename, 1, rc, true, true, regionParser.getRegions(filename), new ArrayList<>(), new ArrayList<>());
}
}
The code above outputs all frame scores of the input audio, this can generate massive amount of output, especially when the audio is long. One good way to trim down the output is to filter out regions with frame scores higher than a preset threshold value. The following shows how this can be done in the @Override section of the callback routine, using a threshold of 0.0.
// Create a callback to handle SAD results from the server
Server.ResultCallback<Scenic.FrameScorerRequest, Scenic.FrameScorerResult> rc = new Server.ResultCallback<Scenic.FrameScorerRequest, Scenic.FrameScorerResult>() {
@Override
public void call(Server.Result<Scenic.FrameScorerRequest, Scenic.FrameScorerResult> r) {
// output frame scores
if (!r.hasError()) {
for (Scenic.FrameScores fs : r.getRep().getResultList()) {
System.out.println(String.format("Received %d frame scores for '%s'", fs.getScoreCount(), fs.getClassId()));
Double[] scores = fs.getScoreList().toArray(new Double[fs.getScoreList().size()]);
int rate = fs.getFrameRate();
// filter to output speech regions with scores > 0.0
int speech_start = 0;
int speech_end = 0;
double sum_scores = 0.0;
int num_frames = 0;
for (int i = 0; i < scores.length; i++) {
if (scores[i] > 0.0)
{
if(speech_end == speech_start)
{
speech_start = i;
}
speech_end = i+1;
sum_scores = sum_scores + scores[i];
num_frames = num_frames + 1;
}
else
{
if(speech_end > speech_start)
{
int start = (int) (100 * speech_start / (double) rate);
int end = (int) (100 * speech_end / (double) rate);
System.out.println(String.format("start: '%d' end: '%d' score:'%f'",start,end,sum_scores/num_frames));
}
speech_start = i;
speech_end = i;
sum_scores = 0.0;
num_frames = 0;
}
}
}
}
System.exit(0);
}
};
SID Enrollment and Scoring Request
This example is a full implementation of a client program which sends an enrollment request to a SID plugin, followed by a scoring request to the same SID plugin.
package com.sri.scenic.api.client;
import com.google.protobuf.InvalidProtocolBufferException;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.*;
import com.sri.scenic.api.client.*;
import org.apache.commons.cli.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.UnsupportedAudioFileException;
import java.io.*;
import java.nio.file.*;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;
public class MySIDEnrollmentAndScore {
private static Logger log = LoggerFactory.getLogger(MySIDEnrollmentAndScore.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "multicond-v1";
private static String pluginName = "sid-embed-v5b";
private static String enrollmentFileName;
private static String speakerName = "Mr.X";
private static String scoreWaveFileName;
private static List<Pair<String, String>> enrollmentOptions = new ArrayList<>();
private static String outputDirName = "./";;
private static RegionParser regionParser = new RegionParser();
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// enrollment file name is passed in as an argument
String enrollmentFileName = args[0];
String scoreWaveFileName = args[1];
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// obtain SID plugin handle
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, null, Scenic.TraitType.CLASS_ENROLLER, pluginList);
// Perform SID enrollment task
MySIDEnrollmentAndScoreRequest(server, pd, speakerName, enrollmentFileName, scoreWaveFileName);
}
public static boolean MySIDEnrollmentAndScoreRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String speakerName, String enrollmentFileName, String scoreWaveFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a callback that handles the enrollment result from the server
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> enrollmentCallback = new Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult>() {
@Override
public void call(Server.Result<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> r) {
// examine enrollment result
if(!r.hasError()){
System.out.println("Enrollment succeeded");
}
else {
log.error("Enrollment request failed: {}", r.getError());
}
}
};
// do enrollment request
// make it a synchronized call, so we know the speaker is enrolled before we make the score request
boolean enrolled = false;
enrolled=ClientUtils.requestEnrollClass(server, pp, speakerName, enrollmentFileName, 0, enrollmentCallback, false, AudioTransferType.SEND_SERIALIZED_BUFFER, new ArrayList<>(), enrollmentOptions);
if(enrolled){
// Create a call back to handle the SID scoring request
Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> scoreCallback = new Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult>() {
@Override
public void call(Server.Result<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> r) {
// do something with the results:
if(!r.hasError()){
log.info("Received {} scores:", r.getRep().getScoreCount());
for(Scenic.GlobalScore gs : r.getRep().getScoreList()){
System.out.println(String.format("speaker{%s} = {%f}",gs.getClassId(),gs.getScore()));
}
}
else{
System.out.println(String.format("Global scorer error: {%s}", r.getError()));
}
System.exit(0);
}
};
// SID is a global scorer, so make a global score reqeust:
return ClientUtils.requestGlobalScore(server, pp, Scenic.TraitType.GLOBAL_SCORER, scoreWaveFileName, 0, scoreCallback, true, AudioTransferType.SEND_SERIALIZED_BUFFER, regionParser.getRegions(scoreWaveFileName), new ArrayList<>(), new ArrayList<>());
}
return false;
}
}
LID Enrollment and Scoring Request
Synchronized Approach
A LID enrollment of a new language may involve multiple enrollment files. The synchronized version of this client waits for the server to complete the enrollment request with each enrollment file sequentially, before requesting LID scores on an input file.
package com.sri.scenic.api.client;
import com.google.protobuf.InvalidProtocolBufferException;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.*;
import com.sri.scenic.api.client.*;
import org.apache.commons.cli.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.UnsupportedAudioFileException;
import java.io.*;
import java.nio.file.*;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
public class MysyncLIDEnrollmentAndScore {
private static Logger log = LoggerFactory.getLogger(MysyncLIDEnrollmentAndScore.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "multi-v1";
private static String pluginName = "lid-embedplda-v1-rc6";
private static String audioList;
private static ArrayList<String> audioFiles = new ArrayList<String>();
private static String audioFilename;
private static int enrollmentFileCount = 0;
private static String languageName = "JKL";
private static String scoreWaveFileName;
private static List<Pair<String, String>> enrollmentOptions = new ArrayList<>();
private static RegionParser regionParser = new RegionParser();
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// enrollment file name is passed in as an argument
String audioList = args[0];
String scoreWaveFileName = args[1];
// read enrollment files from input list
if (!Files.exists(Paths.get(audioList).toAbsolutePath())) {
System.err.println("ERROR: '" + audioList + "' does not exist");
}
BufferedReader br = new BufferedReader(new FileReader(audioList));
while ((audioFilename = br.readLine()) != null){
audioFiles.add(audioFilename);
}
enrollmentFileCount = audioFiles.size();
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// obtain LID plugin handle
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, null, Scenic.TraitType.CLASS_ENROLLER, pluginList);
// Perform LID enrollment task
MyLIDEnrollmentAndScoreRequest(server, pd, languageName, audioFiles, enrollmentFileCount, scoreWaveFileName);
}
public static boolean MyLIDEnrollmentAndScoreRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String languageName, ArrayList<String> audioFiles, int enrollmentFileCount, String scoreWaveFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a callback that handles the enrollment result from the server
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> enrollmentCallback = new Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult>() {
@Override
public void call(Server.Result<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> r) {
// do something with the results:
if(!r.hasError()){
System.err.println("*****Enrollment Succeeded*****");
}
else {
log.error("Enrollment request failed: {}", r.getError());
}
}
};
// do enrollment requests, with one enrollment file at a time
// make a synchronized enrollment call, so we know the language is enrolled before we make the score request
for (String audioFilename : audioFiles) {
boolean enrolled = ClientUtils.requestEnrollClass(server, pp, languageName, audioFilename, 0, enrollmentCallback, false, false, new ArrayList<>(), enrollmentOptions);
if (enrolled) {
// can do something here like print out name of enrollment file used
}
}
// do the score request
// First create a call back to handle the LID scoring result
Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> scoreCallback = new Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult>() {
@Override
public void call(Server.Result<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> r) {
// do something with the results:
if(!r.hasError()){
log.info("Received {} scores:", r.getRep().getScoreCount());
for(Scenic.GlobalScore gs : r.getRep().getScoreList()){
log.info("\t{} = {}", gs.getClassId(), gs.getScore());
}
}
else{
log.error("Global scorer error: {}", r.getError());
}
System.exit(0);
}
};
// LID is a global scorer, so make a global score reqeust:
return ClientUtils.requestGlobalScore(server, pp, Scenic.TraitType.GLOBAL_SCORER, scoreWaveFileName, 0, scoreCallback, true, false, regionParser.getRegions(scoreWaveFileName), new ArrayList<>(), new ArrayList<>());
}
}
Asynchronized Approach
This example provides an alternate LID client example that performs asynchronized enrollment and does not wait for the last enrollment request to complete before making another. Instead, it keeps count on server responses to enrollment requests made, and makes sure that all enrollment requests are completed before making a LID scoring request on an input file.
package com.sri.scenic.api.client;
import com.google.protobuf.InvalidProtocolBufferException;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.*;
import com.sri.scenic.api.client.*;
import org.apache.commons.cli.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.UnsupportedAudioFileException;
import java.io.*;
import java.nio.file.*;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
public class MyasyncLIDEnrollmentAndScore {
private static Logger log = LoggerFactory.getLogger(MyasyncLIDEnrollmentAndScore.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "multi-v1";
private static String pluginName = "lid-embedplda-v1-rc6";
private static String audioList;
private static ArrayList<String> audioFiles = new ArrayList<String>();
private static String audioFilename;
private static int enrollmentFileCount = 0;
private static int responseCount=0;
private static String languageName = "MNO";
private static String scoreWaveFileName;
private static List<Pair<String, String>> enrollmentOptions = new ArrayList<>();
private static String outputDirName = "./";;
private static RegionParser regionParser = new RegionParser();
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// enrollment list is passed in as an argument
String audioList = args[0];
String scoreWaveFileName = args[1];
// read enrollment files from input list and keep count of enrollment files to be processed
if (!Files.exists(Paths.get(audioList).toAbsolutePath())) {
System.err.println("ERROR: '" + audioList + "' does not exist");
}
BufferedReader br = new BufferedReader(new FileReader(audioList));
while ((audioFilename = br.readLine()) != null){
audioFiles.add(audioFilename);
}
enrollmentFileCount = audioFiles.size();
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// obtain LID plugin handle
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, null, Scenic.TraitType.CLASS_ENROLLER, pluginList);
// Perform LID enrollment task
MyLIDEnrollmentAndScoreRequest(server, pd, languageName, audioFiles, enrollmentFileCount, scoreWaveFileName);
}
public static boolean MyLIDEnrollmentAndScoreRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String languageName, ArrayList<String> audioFiles, int enrollmentFileCount, String scoreWaveFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a callback that handles the enrollment result from the server
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> enrollmentCallback = new Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult>() {
@Override
public void call(Server.Result<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> r) {
// do something with the results:
if(!r.hasError()){
System.err.println("*****server processed 1 enrollment file*****");
responseCount++;
}
else {
log.error("Enrollment request failed: {}", r.getError());
}
}
};
for (String audioFilename : audioFiles) {
boolean enrolled = ClientUtils.requestEnrollClass(server, pp, languageName, audioFilename, 0, enrollmentCallback, true, true, new ArrayList<>(), enrollmentOptions);
if (enrolled) {
// can do something here like print out enrollment file processed
}
}
// wait for all enrollments to complete
while(responseCount != enrollmentFileCount) {
try{
TimeUnit.SECONDS.sleep(10);
} catch (InterruptedException e) {
System.out.println("*****Timeout Exception*****");
}
}
// We next do the score request
// First create a call back to handle the LID scoring request
Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> scoreCallback = new Server.ResultCallback<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult>() {
@Override
public void call(Server.Result<Scenic.GlobalScorerRequest, Scenic.GlobalScorerResult> r) {
// do something with the results:
if(!r.hasError()){
log.info("Received {} scores:", r.getRep().getScoreCount());
for(Scenic.GlobalScore gs : r.getRep().getScoreList()){
log.info("\t{} = {}", gs.getClassId(), gs.getScore());
}
}
else{
log.error("Global scorer error: {}", r.getError());
}
System.exit(0);
}
};
// LID is a global scorer, so make a global score reqeust:
return ClientUtils.requestGlobalScore(server, pp, Scenic.TraitType.GLOBAL_SCORER, scoreWaveFileName, 0, scoreCallback, true, true, regionParser.getRegions(scoreWaveFileName), new ArrayList<>(), new ArrayList<>());
}
}
KWS Scoring Request
This portrays a full implementation of a KWS scoring request, against the list of default keywords already trained. Note that unlike the previous examples that requested GlobalScores, this client issues a RegionScoreRequest because KWS is a "RegionScorer".
package com.sri.scenic.api.client;
import com.google.protobuf.InvalidProtocolBufferException;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.*;
import com.sri.scenic.api.client.*;
import org.apache.commons.cli.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.UnsupportedAudioFileException;
import java.io.*;
import java.nio.file.*;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
public class MyKWSRegionScorer {
private static Logger log = LoggerFactory.getLogger(MyKWSRegionScorer.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "r-tdnn-tel-v1";
private static String pluginName = "kws-dynapy-v1";
private static RegionParser regionParser = new RegionParser();
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// audio file name is passed as an argument
String audioFileName = args[0];
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// formulate the frame scoring task request
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, "KWS", Scenic.TraitType.REGION_SCORER, pluginList);
// Perform KWS frame scoring task
MyRegionScoresRequest(server, pd, audioFileName);
}
public static boolean MyRegionScoresRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String filename) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// Create a callback to handle KWS results from the server
Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> rc = new Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult>() {
@Override
public void call(Server.Result<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> r) {
// do something with the results:
if (!r.hasError()) {
log.info("Received {} region scores:", r.getRep().getRegionCount());
for (Scenic.RegionScore rs : r.getRep().getRegionList()) {
log.info("\t{} = {}. From {} to {} ", rs.getClassId(), rs.getScore(), rs.getStartT(), rs.getEndT());
}
} else {
log.error("Region scoring error: {}", r.getError());
}
System.exit(0);
}
};
return ClientUtils.requestRegionScores(server, pp, filename, 0, rc, true, false, regionParser.getRegions(filename), new ArrayList<>(), new ArrayList<>());
}
}
SDD Enrollment And Scoring Request
The following code shows a full implementation of a speaker enrollment request, followed by a scoring request made to a SDD plugin. The scoring request in this client is a RegionScorerRequest because SDD plugins are region scorers.
package com.sri.scenic.api.client;
import com.google.protobuf.InvalidProtocolBufferException;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.*;
import com.sri.scenic.api.client.*;
import org.apache.commons.cli.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.UnsupportedAudioFileException;
import java.io.*;
import java.nio.file.*;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
public class MySDDEnrollmentAndScore {
private static Logger log = LoggerFactory.getLogger(MySDDEnrollmentAndScore.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "multicond-v1";
private static String pluginName = "sdd-sbc-embed-v1-1";
private static String enrollmentFileName;
private static String speakerName = "Mr.X";
private static String scoreWaveFileName;
private static List<Pair<String, String>> enrollmentOptions = new ArrayList<>();
private static RegionParser regionParser = new RegionParser();
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// enrollment file name is passed in as an argument
String enrollmentFileName = args[0];
String scoreWaveFileName = args[1];
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// obtain SDD plugin handle
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, null, Scenic.TraitType.CLASS_ENROLLER, pluginList);
// Perform SDD enrollment task
MySDDEnrollmentAndScoreRequest(server, pd, speakerName, enrollmentFileName, scoreWaveFileName);
}
public static boolean MySDDEnrollmentAndScoreRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String speakerName, String enrollmentFileName, String scoreWaveFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
// First create a callback that handles the enrollment result from the server
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> enrollmentCallback = new Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult>() {
@Override
public void call(Server.Result<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> r) {
// enrollment result received
if(!r.hasError()){
System.out.println("Enrollment succeeded");
}
else {
log.error("Enrollment request failed: {}", r.getError());
}
}
};
// make a synchronized enrollment request, so we know the speaker is enrolled before we make the score request
boolean enrolled = ClientUtils.requestEnrollClass(server, pp, speakerName, enrollmentFileName, 0, enrollmentCallback, false, false, new ArrayList<>(), enrollmentOptions);
if(enrolled){
// do the SDD scoring request
// First create a callback to handle scoring result from server
Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> scoreCallback = new Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult>() {
@Override
public void call(Server.Result<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> r) {
// scoring result received
if (!r.hasError()) {
log.info("Received {} region scores:", r.getRep().getRegionCount());
for (Scenic.RegionScore rs : r.getRep().getRegionList()) {
log.info("\t{} = {}. From {} to {} ", rs.getClassId(), rs.getScore(), rs.getStartT(), rs.getEndT());
}
} else {
log.error("SDD Region scoring error: {}", r.getError());
}
System.exit(0);
}
};
// SDD is a region scorer, so make a region score reqeust:
return ClientUtils.requestRegionScores(server, pp, scoreWaveFileName, 0, scoreCallback, true, false, regionParser.getRegions(scoreWaveFileName), new ArrayList<>(), new ArrayList<>());
}
return false;
}
SAD Supervised Adaptation Request
Below is a full implementation of a SAD request to adapt a new domain from an existing domain. The list of adaptation training files is passed into the client as a file. The code handles both supervised (speech regions specified) and unsupervised (speech regions not specified) SAD adaptations. However, some SAD plugins may not have the unsupervised adaptation capability, in which case the client will exit with a failure message.
package com.sri.scenic.api.client;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Scenic.AnnotationRegion;
import com.sri.scenic.api.Scenic.AudioAnnotation;
import com.sri.scenic.api.Scenic.Domain;
import com.sri.scenic.api.Scenic.Plugin;
import com.sri.scenic.api.Scenic.TraitType;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.ClientUtils;
import com.sri.scenic.api.utils.LearningParser;
import com.sri.scenic.api.utils.LearningParser.LearningDataType;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import javax.sound.sampled.UnsupportedAudioFileException;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.CommandLineParser;
import org.apache.commons.cli.DefaultParser;
import org.apache.commons.cli.HelpFormatter;
import org.apache.commons.cli.Option;
import org.apache.commons.cli.Options;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MySADAdaptation {
private static Logger log = LoggerFactory.getLogger(MySADAdaptation.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "tel-v1";
private static String newDomainName = "new-tel-v1";
private static String pluginName = "sad-dnn-v6_rc5";
private static LearningParser learningParser = new LearningParser();
private static LearningDataType dataType;
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// audio file list is passed as an argument
String audioFileList = args[0];
learningParser.parse(audioFileList);
if (!learningParser.isValid()) {
System.err.println("Invalid input file: " + audioFileList);
System.exit(-1);
}
if (learningParser.hasRegions()) {
dataType = LearningDataType.SUPERVISED_WITH_REGIONS;
} else if (learningParser.hasClasses()) {
dataType = LearningDataType.SUPERVISED;
} else {
dataType = LearningDataType.UNSUPERVISED;
}
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// find plugin handle for adaptation task
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, "SAD", learningParser.isUnsupervised() ? Scenic.TraitType.UNSUPERVISED_ADAPTER : Scenic.TraitType.SUPERVISED_ADAPTER, pluginList);
// Preproces audio - doesn't matter if supervised or unsupervised
String adaptID = UUID.randomUUID().toString();
Plugin p = pd.getFirst();
Domain d = pd.getSecond();
// optional annotations, generated if found in the parser (supervised adaptation)
Map<String, List<AudioAnnotation>> annotations = new HashMap<>();
// annotations -> <classID> ->* <AudioAnnotations>, annotations will be empty for unsupervised adaptation
int numPreprocessed = OliveLearn.preprocessAllAudio(server, p, d, learningParser, adaptID, annotations);
if(!learningParser.isUnsupervised())
{
// supervised adaptation
if (numPreprocessed > 0) {
OliveLearn.finalizeSupervisedAdaptation(server, p, d, adaptID, newDomainName, annotations);
}
else {
System.err.println("Can not adapt domain because all audio preprocessing attempts failed.");
}
}
else {
// unsupervised adaptation
if (numPreprocessed > 0) {
OliveLearn.finalizeUnsupervisedAdaptation(server, p, d, adaptID, newDomainName);
}
else {
System.err.println("Can not adapt domain because all audio preprocessing attempts failed.");
}
}
System.out.println("");
System.out.println("Learning finished. Exiting...");
System.exit(0);
}
}
TPD Enrollment and Scoring Request
This final example shows a full implementation of a TPD topic enrollment request using positive examples from selected regions of an audio, followed by a scoring request of the enrolled topic on another audio. Enrollment using negative audio samples are also included in this example as commented out lines.
package com.sri.scenic.api.client;
import com.google.protobuf.InvalidProtocolBufferException;
import com.sri.scenic.api.Scenic;
import com.sri.scenic.api.Server;
import com.sri.scenic.api.utils.*;
import com.sri.scenic.api.client.*;
import org.apache.commons.cli.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sound.sampled.UnsupportedAudioFileException;
import java.io.*;
import java.nio.file.*;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;
public class MyTPDEnrollmentAndScore {
private static Logger log = LoggerFactory.getLogger(MyTPDEnrollmentAndScore.class);
private static final int TIMEOUT = 10000;
private static final String DEFAULT_SERVERNAME = "localhost";
private static final int DEFAULT_PORT = 5588;
private static String domainName = "r-tdnn-tel-v1";
private static String pluginName = "tpd-dynapy-v1";
private static String posEnrollmentFileName;
private static String negEnrollmentFileName;
private static String topicName = "hello";
private static String scoreWaveFileName;
private static List<Pair<String, String>> enrollmentOptions = new ArrayList<>();
//private static String outputDirName = "./";;
private static RegionParser regionParser = new RegionParser();
/**
* Main execution point
*
* @throws Exception if there was an error
*/
public static void main(String[] args) throws Exception {
// enrollment file name is passed in as an argument
String posEnrollmentFileName = args[0];
String negEnrollmentFileName = args[1];
String scoreWaveFileName = args[2];
// Setup the connection to the (scenic) server
Server server = new Server();
server.connect("scenic-ui", DEFAULT_SERVERNAME,
DEFAULT_PORT,
DEFAULT_PORT + 1,
TIMEOUT); // may need to adjust timeout
// wait for the connection
long start_t = System.currentTimeMillis();
while (!server.getConnected().get() && System.currentTimeMillis() - start_t < TIMEOUT) {
try {
synchronized (server.getConnected()) {
server.getConnected().wait(TIMEOUT);
}
} catch (InterruptedException e) {
// Keep waiting
}
}
// report if connection fails
if (!server.getConnected().get()) {
log.error("Unable to connect to the SCENIC server: {}", DEFAULT_SERVERNAME);
throw new Exception("Unable to connect to server");
}
// set audio mode to read from buffer
//Scenic.Audio audio = SimpleClient.packageAudioAsRawBuffer(enrollmentFileName, 0, null);
// ask the server for a list of current plugins
List<Pair<Scenic.Plugin, Scenic.Domain>> pluginList = SimpleClient.requestPlugins(server, true, true);
// obtain TPD plugin handle
Pair<Scenic.Plugin, Scenic.Domain> pd = ClientUtils.findPluginDomainByTrait(pluginName, domainName, null, Scenic.TraitType.CLASS_ENROLLER, pluginList);
// Perform TPD enrollment and scoring tasks
MyTPDEnrollmentAndScoreRequest(server, pd, topicName, posEnrollmentFileName, negEnrollmentFileName, scoreWaveFileName);
}
public static boolean MyTPDEnrollmentAndScoreRequest(Server server, Pair<Scenic.Plugin, Scenic.Domain> pp, String speakerName, String posEnrollmentFileName, String negEnrollmentFileName, String scoreWaveFileName) throws ClientException, IOException, UnsupportedAudioFileException {
if (null == pp) {
return false;
}
//enroll a positive example
List<RegionWord> posRegions = new ArrayList<>();
posRegions.add(new RegionWord(0.100, 2.500)); // NOTE to add regions in seconds
posRegions.add(new RegionWord(7.500, 10.000)); // NOTE to add regions in seconds
/**
//we can also enroll a negative example
enrollmentOptions.add(new Pair<>("isNegative", "True"));
List<RegionWord> negRegions = new ArrayList<>();
negRegions.add(new RegionWord(1.000, 1.500)); // NOTE to add regions in seconds
negRegions.add(new RegionWord(5.000, 7.000)); // NOTE to add regions in seconds
**/
// First create a callback that handles the enrollment result from the server
Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> enrollmentCallback = new Server.ResultCallback<Scenic.ClassModificationRequest, Scenic.ClassModificationResult>() {
@Override
public void call(Server.Result<Scenic.ClassModificationRequest, Scenic.ClassModificationResult> r) {
// examine enrollment result
if(!r.hasError()){
System.out.println("Enrollment succeeded");
}
else {
log.error("Enrollment request failed: {}", r.getError());
}
}
};
// do enrollment request
// make it a synchronized call, so we know enrollment is complete before we score
boolean enrolled = false;
enrolled=ClientUtils.requestEnrollClass(server, pp, topicName, posEnrollmentFileName, 0, enrollmentCallback, false, false, posRegions, enrollmentOptions);
/**
//goes with negative example
enrolled=ClientUtils.requestEnrollClass(server, pp, topicName, negEnrollmentFileName, 0, enrollmentCallback, false, false, posRegions, enrollmentOptions);
**/
if(enrolled){
// Create a call back to handle the TPD scoring request
Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> scoreCallback = new Server.ResultCallback<Scenic.RegionScorerRequest, Scenic.RegionScorerResult>() {
@Override
public void call(Server.Result<Scenic.RegionScorerRequest, Scenic.RegionScorerResult> r) {
// do something with the results:
if(!r.hasError()){
log.info("Received {} regions:", r.getRep().getRegionCount());
for(Scenic.RegionScore rs : r.getRep().getRegionList()){
log.info("\t{} = {}, From {} to {}", rs.getClassId(), rs.getScore(), rs.getStartT(), rs.getEndT());
}
}
else{
log.error("Region scorer error: {}", r.getError());
}
System.exit(0);
}
};
// TPD is a region scorer, so make a region score reqeust
return ClientUtils.requestRegionScores(server, pp, scoreWaveFileName, 0, scoreCallback, true, false, regionParser.getRegions(scoreWaveFileName), new ArrayList<>(), new ArrayList<>());
}
return false;
}
}