Abstract:The goal of this work is to efficiently identify visually similar patterns in images, e.g. identifying an artwork detail copied between an engraving and an oil painting, or recognizing parts of a night-time photograph visible in its daytime counterpart. Lack of training data is a key challenge for this co-segmentation task. We present a simple yet surprisingly effective approach to overcome this difficulty: we generate synthetic training pairs by selecting segments in an image and copy-pasting them into another image. We then learn to predict the repeated region masks. We find that it is crucial to predict the correspondences as an auxiliary task and to use Poisson blending and style transfer on the training pairs to generalize on real data. We analyse results with two deep architectures relevant to our joint image analysis task: a transformer-based architecture and Sparse Nc-Net, a recent network designed to predict coarse correspondences using 4D convolutions. We show our approach provides clear improvements for artwork details retrieval on the Brueghel dataset and achieves competitive performance on two place recognition benchmarks, Tokyo247 and Pitts30K. We also demonstrate the potential of our approach for unsupervised image collection analysis by introducing a spectral graph clustering approach to object discovery and demonstrating it on the object discovery dataset of \cite{rubinstein2013unsupervised} and the Brueghel dataset. Our code and data are available at <a class="link-external link-http" href="http://imagine.enpc.fr/~shenx/SegSwap/" rel="external noopener nofollow">this http URL</a>.

Extensible Image Object Co-Segmentation with Sparse Cooperative Relations

Weak Supervision Learning for Object Co-Segmentation

Scalable Image Co-Segmentation Using Color and Covariance Features

Coarse-to-fine Image Co-segmentation with Intra and Inter Rank Constraints

Object Co-Segmentation Via Salient and Common Regions Discovery

Cosegmentation and Cosketch by Unsupervised Learning.

Unsupervised Co-segmentation of Complex Image Set via Bi-harmonic Distance Governed Multi-level Deformable Graph Clustering

Object Co-Segmentation Via Discriminative Low Rank Matrix Recovery.

Robust Object Co-Segmentation Using Background Prior.

Human Centred Object Co-Segmentation

Toward Stable Co-Saliency Detection and Object Co-Segmentation

Object co-segmentation via weakly supervised data fusion

Image Co-segmentation Via Multi-scale Local Shape Transfer

Robust Video Object Cosegmentation.

Deep Object Co-segmentation via Spatial-Semantic Network Modulation

Mutual Learning Between Saliency and Similarity: Image Cosegmentation via Tree Structured Sparsity and Tree Graph Matching.

Object Co-skeletonization with Co-segmentation

Cooperative Object Search and Segmentation in Internet Images

Learning Co-segmentation by Segment Swapping for Retrieval and Discovery

Group-wise Deep Object Co-Segmentation with Co-Attention Recurrent Neural Network

Image Co-skeletonization via Co-segmentation