Abstract:Deception detection is an important task that has been a hot research topic due to its potential applications. It can be applied in many areas, from national security (e.g., airport security, jurisprudence, and law enforcement) to real-life applications (e.g., business and computer vision). However, some critical problems still exist and are worth more investigation. One of the significant challenges in the deception detection tasks is the data scarcity problem. Until now, only one multi-modal benchmark open dataset for human deception detection has been released, which contains 121 video clips for deception detection (i.e., 61 for deceptive class and 60 for truthful class). Such an amount of data is hard to drive deep neural network-based methods. Hence, those existing models often suffer from overfitting problems and low generalization ability. Moreover, the ground truth data contains some unusable frames for many factors. However, most of the literature did not pay attention to these problems. Therefore, in this paper, we design a series of data preprocessing methods to deal with the aforementioned problem first. Then, we propose a multi-modal deception detection framework to construct our novel emotional state-based feature and use the open toolkit openSMILE to extract the features from the audio modality. We also design a voting scheme to combine the emotional states information obtained from visual and audio modalities. Finally, we can determine the novel emotion state transformation feature with our self-designed algorithms. In the experiment, we conduct the critical analysis and comparison of the proposed methods with the state-of-the-art multi-modal deception detection methods. The experimental results show that the overall performance of multi-modal deception detection has a significant improvement in the accuracy from 87.77% to 92.78% and the ROC-AUC from 0.9221 to 0.9265.

Can earnings conference calls tell more lies? A contrastive multimodal dialogue network for advanced financial statement fraud detection

Which Spoken Language Markers Identify Deception in High-Stakes Settings? Evidence From Earnings Conference Calls

Fraud detection in telephone conversations for financial services using linguistic features

Attentive statement fraud detection: Distinguishing multimodal financial data with fine-grained attention

Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks

Tracking disclosure change trajectories for financial fraud detection

Leveraging Financial Social Media Data for Corporate Fraud Detection

Multimodal and Contrastive Learning for Click Fraud Detection

DialogueGAT: A Graph Attention Network for Financial Risk Prediction by Modeling the Dialogues in Earnings Conference Calls.

Shing: A Conversational Agent to Alert Customers of Suspected Online-payment Fraud with Empathetical Communication Skills

Fine-Grained Question-Level Deception Detection Via Graph-Based Learning and Cross-Modal Fusion

An Analysis on Financial Statement Fraud Detection for Chinese Listed Companies Using Deep Learning

Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues

Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learning

Analysis, Evaluation, and Future Directions on Multimodal Deception Detection

A Semi-Supervised Speech Deception Detection Algorithm Combining Acoustic Statistical Features and Time-Frequency Two-Dimensional Features

Unearthing Financial Statement Fraud: Insights from News Coverage Analysis

Deception detection using multimodal fusion approaches

Cooperative Fraud Detection Model with Privacy-Preserving in Real CDR Datasets.

How to Detect and Forecast Corporate Fraud by Media Reports? an Approach Using Machine Learning and Qualitative Comparative Analysis

Constructing Robust Emotional State-based Feature with a Novel Voting Scheme for Multi-modal Deception Detection in Videos