Abstract:Advances in multimedia data acquisition and storage technology have led to the growth of very large multimedia databases. Analyzing this huge amount of multimedia data to discover useful knowledge is a challenging problem. This challenge has opened the opportunity for research in Multimedia Data Mining (MDM). Multimedia data mining can be defined as the process of finding interesting patterns from media data such as audio, video, image and text that are not ordinarily accessible by basic queries and associated results. The motivation for doing MDM is to use the discovered patterns to improve decision making. MDM has therefore attracted significant research efforts in developing methods and tools to organize, manage, search and perform domain specific tasks for data from domains such as surveillance, meetings, broadcast news, sports, archives, movies, medical data, as well as personal and online media collections. This paper presents a survey on the problems and solutions in Multimedia Data Mining, approached from the following angles: feature extraction, transformation and representation techniques, data mining techniques, and current multimedia data mining systems in various application domains. We discuss main aspects of feature extraction, transformation and representation techniques. These aspects are: level of feature extraction, feature fusion, features synchronization, feature correlation discovery and accurate representation of multimedia data. Comparison of MDM techniques with state of the art video processing, audio processing and image processing techniques is also provided. Similarly, we compare MDM techniques with the state of the art data mining techniques involving clustering, classification, sequence pattern mining, association rule mining and visualization. We review current multimedia data mining systems in detail, grouping them according to problem formulations and approaches. The review includes supervised and unsupervised discovery of events and actions from one or more continuous sequences. We also do a detailed analysis to understand what has been achieved and what are the remaining gaps where future research efforts could be focussed. We then conclude this survey with a look at open research directions.

Guest Editorial: Ad Hoc Web Multimedia Analysis with Limited Supervision

Multi-modal Deep Analysis for Multimedia

Annotation Efficient Cross-Modal Retrieval with Adversarial Attentive Alignment

Guest editorial: web multimedia semantic inference using multi-cues

Assistive tagging: A survey of multimedia tagging with human-computer joint exploration

Current Research Status and Prospects on Multimedia Content Understanding

Multimedia Analysis with Deep Learning

Assistive tagging

Adversarial Training Towards Robust Multimedia Recommender System

A Semantic Approach to Recommending Text Advertisements for Images

IEEE Access Special Section Editorial: Intelligent Sensing on Mobile and Social Media Analytics

Semantic Link Network-Based Model for Organizing Multimedia Big Data

Multimedia data mining: state of the art and challenges

Self-Supervised Multimodal Learning: A Survey

Discriminating Joint Feature Analysis for Multimedia Data Understanding

Learning Multimodal Latent Attributes

Effective and Efficient Multi-Facet Web Image Annotation

Multimedia Analysis and Retrieval System

Large-Scale Video Understanding with Limited Training Labels

Multimedia search reranking: A literature survey