Abstract:Data mining can hardly solve but always faces a problem that there is little meaningful information within the dataset serving a given requirement. Faced with multiple unknown datasets, to allocate data mining resources to acquire more desired data, it is necessary to establish a data quality assessment framework based on the relevance between the dataset and requirements. This framework can help the user to judge the potential benefits in advance, so as to optimize the resource allocation to those candidates. However, the unstructured data (e.g., image data) often presents dark data states, which makes it tricky for the user to understand the relevance based on content of the dataset in real time. Even if all data have label descriptions, how to measure the relevance between data efficiently under semantic propagation remains an urgent problem. Based on this, we propose a Deep Hash-based Relevance-aware Data Quality Assessment framework, which contains off-line learning and relevance mining parts as well as an on-line assessing part. In the off-line part, we first design a Graph Convolution Network (GCN)-AutoEncoder hash (GAH) algorithm to recognize the data (i.e., lighten the dark data), then construct a graph with restricted Hamming distance, and finally design a Cluster PageRank (CPR) algorithm to calculate the importance score for each node (image) so as to obtain the relevance representation based on semantic propagation. In the on-line part, we first retrieve the importance score by hash codes and then quickly get the assessment conclusion in the importance list. On the one hand, the introduction of GCN and co-occurrence probability in the GAH promotes the perception ability for dark data. On the other hand, the design of CPR utilizes hash collision to reduce the scale of graph and iteration matrix, which greatly decreases the consumption of space and computing resources. We conduct extensive experiments on both single-label and multi-label datasets to assess the relevance between data and requirements as well as test the resources allocation. Experimental results show our framework can gain the most desired data with the same mining resources. Besides, the test results on Tencent1M dataset demonstrate the framework can complete the assessment with a stability for given different requirements.

Learning hash codes for efficient content reuse detection.

Online latent semantic hashing for cross-media retrieval.

Efficient Discrete Supervised Hashing for Large-scale Cross-modal Retrieval

Supervised Hashing With Pseudo Labels For Scalable Multimedia Retrieval

Compact scalable hash from deep learning features aggregation for content de-duplication

A Mixed Generative-Discriminative Based Hashing Method

Feature aggregating hashing for image copy detection

On Real-Time Detecting Duplicate Web Videos

Continuous Word Embeddings For Detecting Local Text Reuses At The Semantic Level

Efficient Image Hashing with Geometric Invariant Vector Distance for Copy Detection

Multiple Code Hashing for Efficient Image Retrieval

Perceptual Robust Hashing for Video Copy Detection with Unsupervised Learning.

Robust Hashing via Global and Local Invariant Features for Image Copy Detection

Large-Scale Duplicate Detection for Web Image Search

Probabilistic Hypergraph Based Hash Codes for Social Image Search

Deep Hash-based Relevance-aware Data Quality Assessment for Image Dark Data.

TDHPPIR: An Efficient Deep Hashing Based Privacy-Preserving Image Retrieval Method

A Revisit on Deep Hashings for Large-scale Content Based Image Retrieval

Efficient Supervised Graph Embedding Hashing for large-scale cross-media retrieval

Sequential Discrete Hashing for Scalable Cross-Modality Similarity Retrieval

Collaborative Learning for Extremely Low Bit Asymmetric Hashing