Abstract:The rapid development of social network and online multimedia technology makes it possible to address traditional image and video enhancement problems, with the aid of online similar reference data. In this paper, we tackle the problem of super-resolution (SR) in this way, specifically aiming to handle the “one-to-many” problem between the image patches of low resolution (LR) and high resolution (HR). We propose a manifold localized deep external compensation (MALDEC) network to additionally utilize reference images, i. e., retrieved similar images in cloud database and reference HR frame in a video, to provide an accurate localization and mapping to the HR manifold, and compensate the lost high-frequency details. The proposed network employs a three-step recovery: 1) internal structure inference, which uses the LR image itself and the internally inferred high frequency information to preserve main structure of the HR image; 2) manifold localization, which localizes the HR manifold and constructs the correspondence between the internal inferred image and the external images; and 3) external compensation, which introduces the external references of retrieved similar patches based on manifold localization information to reconstruct the high-frequency details. The learnable components of MALDEC, internal structure inference, and external compensation, are trained jointly to make a good tradeoff between these two terms for an optimal SR result. Finally, the proposed method is examined under three tasks: cloud-based image SR, multi-pose face reconstruction, and reference frame-guided video SR. Extensive experiments demonstrate the superiority of our method than the state-of-the-art SR methods in both objective and subjective evaluations, and our method offers new state-of-the-art performance.

Aligning Coupled Manifolds for Face Hallucination

Learning Visually Aligned Semantic Graph for Cross-Modal Manifold Matching.

Coupled discriminative manifold alignment for low-resolution face recognition

Face Alignment Based on 3D Face Shape Model and Markov Random Field.

Learning Adaptive Local Distance Metric For Face Hallucination

Hallucinating Faces: Global Linear Modal Based Super-Resolution and Position Based Residue Compensation

Face Super-Resolution Via Bilayer Contextual Representation

Face Hallucination by Learning Local Distance Metric.

The face image super-resolution algorithm based on combined representation learning

Manifold Alignment Via Corresponding Projections.

Hallucinating 3d Faces

Contextual-Field Supported Iterative Representation for Face Hallucination.

Hallucinating Faces: LPH Super-Resolution and Neighbor Reconstruction for Residue Compensation.

Super-resolution for facial image using multilateral affinity function

Reference-Guided Deep Super-Resolution Via Manifold Localized External Compensation

Collaborative reconstruction-based manifold-manifold distance for face recognition with image sets

Position-based face hallucination method

Face Hallucination by Attentive Sequence Optimization with Reinforcement Learning

Face Hallucination Via Semi-Kernel Partial Least Squares

Super-resolution of human face image using canonical correlation analysis

Locality-Constrained Double Low-Rank Representation for Effective Face Hallucination