sum-txt-llm-commercial (Summarization of text inputs)

Version Changelog

Plugin Version	Change
v1.0.0	Initial plugin release with OLIVE 6.1.0.

Description

This Text Summarization plugin creates a short summary of the provided text input(s). It transforms raw text inputs into a short paragraph with the aim of preserving key information contained within the input text.

This plugin performs this task using a large language model (LLM) external to the plugin, either run in OLIVE or hosted separately.

LLMs have a max. context length that they can use for processing inputs (prompts) and outputs. Long texts that do not fit in the LLM's context window are therefore processed by splitting them into overlapping sequences of words. Each of these sequences are then summarized using the LLM. Subsequently, semantic embeddings are then generated from each text summary. Clusters are then built from the summary embeddings, and high-level summaries are created for each cluster using the LLM. Finally, an overall summary is created from the high-level summaries using the LLM.

Domains

• multi-v1
  • Uses an LLM to create a summary of the input text. The accuracy as well as language support varies between different LLMs.

Inputs

A text string or text-populated file to process.

An example input text, in English:

Shortly after 9 A.M. local time on Friday morning, thousands of hot dogs spilled onto Interstate 83 in York County, Pennsylvania following a crash involving a tractor trailer and two other vehicles. Several people were injured in the incident and were treated at York Hospital, although authorities later said that those were not life threatening.

The tractor trailer split open after a mechanical issue resulted in the truck brushing against a concrete divider, resulting in boxes of frozen hotdogs spilling out. According to the owner of a business located near the crash site: "We heard a loud bang, and then we heard about three more loud bangs. We knew it was a tire first, blowout here on 83. But then after that, we knew there was some other type of accident because it went on for a minute or two."

The wieners were smashed along the road. A photo posted by Shrewsbury Volunteer Fire Company showed a damaged concrete divider in the incident. The spill stopped traffic in both directions, and commutes were delayed by "at least a half an hour," according to The Independent. The lanes reopened later that afternoon.

Cleanup efforts extended into the late morning hours, and allegedly involved rakes. Brad Dauberman, who serves as Shrewsbury, Pennsylvania fire company chief described the cleanup process: "Once those [dogs] leave the truck and hit the road, that’s all garbage, and it’s still pretty warm. [...] I can tell you personally, hot dogs are very slippery. I did not know that." Dauberman also stated that in order to complete the task, they were required to use a front-end loader in order to scoop up clods of meat to be jettisoned into a dump truck.

In total, over one thousand pounds of hotdogs were lost in the spill. The Pennsylvania State Police were investigating the incident.

Alternatively, outputs from an ASR plugin and an SDD plugin can be passed directly to the plugin via the optional parameter 'asr_regions' and 'speaker_regions', respectively, in a workflow to generate a conversation summary.

Outputs

The output format for Summarization plugins is simply text.

An example output string generated from the input text above:

A traffic incident in York County, Pennsylvania, caused thousands of hot dogs to spill onto Interstate 83. The accident involved a tractor-trailer, two other vehicles, and a mechanical failure. Several individuals sustained injuries and received treatment at York Hospital. Cleanup efforts, including the use of a front-end loader, lasted for several hours and resulted in over one thousand pounds of hot dogs being discarded. The Pennsylvania State Police are currently investigating the cause of the incident.

Functionality (Traits)

The functions of this plugin are defined by its Traits and implemented API messages. A list of these Traits is below, along with the corresponding API messages for each. Click the message name below to be brought to additional implementation details below.

• CompositeScorer – Plugin accepts and analyzes one or multiple inputs, and outputs a combination of known Scorer result types as output. In the case of the text summarization plugin, texts are provided as inputs, with the expectation that the output will be text strings with concise summaries of the input texts. If a transcript from an ASR plugin is provided directly via the optional parameter asr_regions in a workflow, the plugin will summarize the conversation from the transcript and will return the summary. Optionally, speaker labels from an SDD plugin can be provided via the optional parameter speaker_regions in a workflow, which the plugin can take into account when creating a summary.
  • CompositeScorerRequest
  • CompositeScorerResult
    • TextTransformationResult

Compatibility

OLIVE 6.1+

Limitations

This plugin is based on an LLM. Its performance critically depends therefore on the LLM's performance on the task, in particular for low-resource languages. There is often a correlation between the number of parameters of an LLM and its performance on complex tasks. Hence, larger LLMs tend to perform better. However, they also tend to use more resources and have lower processing speed given the same hardware.

We tested this plugin using Google's Gemma-3-4B-it-qat-q4_0-gguf and Gemma-3-12B-it-qat-q4_0-gguf. Performance with other LLMs may vary.

Comments

Large Language Model (LLM) Required

This plugin relies on a Large Language Model (LLM) for the heavy lifting of its task. This plugin can only be used with an appropriately configured OLIVE server that has been started with the LLM server active. See the LLM Configuration Documentation for more information, and refer to the Martini documentation to make sure the appropriate startup procedure is followed.

GPU Support

Please refer to the OLIVE GPU Installation and Support documentation page for instructions on how to enable and configure GPU capability in supported plugins. By default this plugin will run on CPU only; however, the speed of the embedding model computation used for long text inputs is greatly enhanced when using GPU.

Text Transformation Options

The following region scoring options are available to this plugin, adjustable in the plugin's configuration file; plugin_config.py.

Option Name	Description	Default	Expected Range
llm_base_url	LLM base URL of an OpenAI API compatible LLM server (such as llama-server, vLLM, or hosted LLMs such as OpenAI).	http://127.0.0.1:5007/v1
model	Name of the LLM model to use.	gemma3-4b
api_key	API key/token to use for the LLM server, if required.	token
asr_threshold	Threshold for using input words in optional 'asr_regions' input. (Only relevant for ingesting ASR region outputs directly from an ASR plugin in a workflow.)	0.0
llm_ctx_window	LLM context window length, i.e., the number of tokens in the context window of the LLM. By default, the plugin queries the LLM server for the max. LLM context window. This setting is used as fallback if the query fails.	8192

If you find this plugin to not perform adequately for your data conditions, or have a specific use case, please get in touch with SRI to discuss how the plugin can be tuned for optimal performance on your data.