dfv-facetransformer-commercial (Deepfake Video Detection)
Version Changelog
| Plugin Version | Change |
|---|---|
| v1.0.0 | Initial plugin, released with OLIVE 6.1.0 |
Description
Deepfake video detection (DFV) plugins process an input video and classify whether detected faces in the image were deepfakes or bonafide human faces. Such face deepfakes can be instances of artificial manipulation or could have been entirely synthetically generated by a variety of generative AI algorithms based on generative adversarial networks (GANs), diffusion, or other approaches. Detecting synthetically generated faces in videos is crucial to preventing misinformation, and safeguarding against malicious activities such as impersonation and fraud.
This plugin detects whether faces in a given video came from a photo or from a synthetic system. The plugin first uses face detection to locate faces in frames of the video and then uses a deepfake detection model based on Swin transformers to compute a score for each detected face in an input video. The scores are subsequently transformed to a likelihood ratio using linear calibration. It is adapted from the model described in Xu, Y., Liang, J., Jia, G., Yang, Z., Zhang, Y., & He, R. (2023). TALL: Thumbnail layout for deepfake video detection. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 22658-22668).
Domains
- multi-v1
- A general purpose face deepfake video detection domain.
Inputs
A video file to process.
Outputs
Deepfake video detection (DFV) plugins are 'bounding box' scorers - the output of a video-processing bounding box scorer is a class (in this case "synthetic" representing a deepfake face), a corresponding score, an associated start and end timestamp denoting when the bounding box is valid, and 4 points localizing the detected face, time, and score that attempt to localize a deepfake face within the video frame. The detection scores represent whether a face in a given frame of a video is detected as being artificially manipulated/synthetically generated or not. The scores are log-likelihood ratios where a score of greater than "0" is considered a "synthetic" or face deepfake detection and a score below "0" is considered an image of a face from a bonafide photo.
That output looks like this:
<file> synthetic <score> (<x1>, <y1>, <x2>, <y2>) (<start_seconds>, <end_seconds>)
Where the bounding box represents the location of a face and is defined by the four coordinates in parentheses:
(Upper Left: x1, y1 | Lower Right: x2, y2)
Note that the bounding box only indicates a detected face and the associated score indicates whether it is likely a deepfake or not.
An example output could look like this:
input-video-1.mp4 synthetic -1.9423 (154, 78, 657, 745) (10.40, 12.55)
input-video-2.mkv synthetic 1.2817 (369, 158, 597, 478) (978.31, 978.84)
Functionality (Traits)
The functions of this plugin are defined by its Traits and implemented API messages. A list of these Traits is below, along with the corresponding API messages for each. Click the message name below to go to additional implementation details below.
- BOUNDING_BOX_SCORER – Score all submitted images or videos, returning labeled bounding box regions within the image frames, or labeled bounding box regions with an associated start and end time region if scoring video files.
Compatibility
OLIVE 6.1+
Limitations
The deepfake face detection model was trained on photorealistic real and synthetic images of faces from videos. It is not designed to detect deepfakes of other objects or drawings of faces.
Comments
Detecting deepfake videos is a cat-and-mouse game. The field is (as of writing) moving incredibly fast with new advances and models constantly released. We strove to adopt a well-trained public deepfake detection model but cannot guarantee that it will be able to detect novel or recently developed generators.
GPU Support
Please refer to the OLIVE GPU Installation and Support documentation page for instructions on how to enable and configure GPU capability in supported plugins. By default this plugin will run on CPU only.
Global Options
This plugin does not currently have user-configurable options, though it is possible for some performance tweaks and configuration changes to be made. If you find this plugin to not perform adequately for your data conditions, or have a specific use case, please get in touch with SRI to discuss how the plugin can be tuned for optimal performance on your data.