Abstract:With the development of network technology, the number of gambling websites has grown dramatically, causing a threat to social stability. There are many machine learning-based methods are proposed to identify gambling websites by analyzing the URL, the text, and the images of the websites. Nevertheless, most of the existing methods ignore one important piece of information, i.e., the text within the website images. Only the visual features of images are extracted for detection, while the semantic features of texts on the images are ignored. However, these texts have key information clearly pointing to gambling websites, which can help us identify such websites more accurately. Therefore, how to fuse image and text multimodal data is a challenge that should be met.Motivated by this, in this paper, we propose a hybrid multimodal data fusion-based method for identifying gambling websites by extracting and fusing visual and semantic features of the website screenshots. First, we fine tune the pretrained ResNet34 model to train an image classifier and to extract visual features of webpage screenshots. Second, we extract textual content from webpage screenshots through the optical character recognition (OCR) technique. We use pretrained Word2Vec word vectors as the initial embedding layer and use Bi-LSTM to train a text classifier and extract semantic features of textual content on the screenshots. Third, we use self-attention to fuse the visual and semantic features and train a multimodal classifier. The prediction results of image, text, and multimodal classifiers are fused by the late fusion method to obtain the final prediction result. To demonstrate the effectiveness of the proposed method, we conduct experiments on the webpage screenshot dataset we collected. The experimental results indicate that OCR text on the webpage screenshots has strong semantic features and the proposed hybrid multimodal data fusion based method can effectively improve the performance in identifying gambling websites, with accuracy, precision, recall, and F1-score all over 99%.

Classification of Forms with Similar Layouts Based on Mixed Gaussian Weighted Mask

A Novel Image Classification Method Based on Manifold Learning and Gaussian Mixture Model

Face Recognition Based on Local Binary Pattern and Improved Pairwise-constrained Multiple Metric Learning

Gabor Wavelets Transform and Extended Nearest Feature Space Classifier for Face Recognition

Key-Guided Identity Document Classification Method by Graph Attention Network

Face Recognition Based on LDA and Improved Pairwise-Constrained Multiple Metric Learning Method.

Classification of Facial Images Using Gaussian Mixture Models.

Multi-view Gender Classification Based on Local Gabor Binary Mapping Pattern and Support Vector Machines.

Large-margin multi-view Gaussian process for image classification

Improved Local Binary Patterns For Classification Of Masses Using Mammography

A Hybrid Multimodal Data Fusion-Based Method for Identifying Gambling Websites

A new automatic method for mass detection in mammography with false positives reduction by supported vector machine

A Benign and Malignant Mass Classification Algorithm Based on an Improved Level Set Segmentation and Texture Feature Analysis

A multi-manifold semi-supervised Gaussian mixture model for pattern classification

Weighted Contourlet Binary Patterns and Image-Based Fisher Linear Discriminant for Face Recognition.

Grassmannian Discriminant Maps (GDM) for Manifold Dimensionality Reduction with Application to Image Set Classification

Description and recognition of form and automated form data entry.

Probabilistic Classification by Density Estimation Using Gaussian Mixture Model and Masked Autoregressive Flow

Stripe Model: An Efficient Method to Detect Multi-form Stripe Structures.

GroupLink: An End-to-end Multitask Method for Word Grouping and Relation Extraction in Form Understanding

Image classification based on weighted nonconvex low-rank and discriminant least squares regression