Abstract:Iconclass is an iconographic thesaurus which is widely used in the digital heritage domain to describe subjects depicted in artworks. Each subject is assigned a unique descriptive code, which has a corresponding textual definition. The assignment of Iconclass codes is a challenging task for computational systems, due to the large number of available labels in comparison to the limited amount of training data available. Transfer learning has become a common strategy to overcome such a data shortage. In deep learning, transfer learning consists in fine-tuning the weights of a deep neural network for a downstream task. In this work, we present a deep retrieval framework which can be fully fine-tuned for the task under consideration. Our work is based on a recent approach to this task, which already yielded state-of-the-art performance, although it could not be fully fine-tuned yet. This approach exploits the multi-linguality and multi-modality that is inherent to digital heritage data. Our framework jointly processes multiple input modalities, namely, textual and visual features. We extract the textual features from the artwork titles in multiple languages, whereas the visual features are derived from photographic reproductions of the artworks. The definitions of the Iconclass codes, containing useful textual information, are used as target labels instead of the codes themselves. As our main contribution, we demonstrate that our approach outperforms the state-of-the-art by a large margin. In addition, our approach is superior to the M 3 P feature extractor and outperforms the multi-lingual CLIP in most experiments due to the better quality of the visual features. Our out-of-domain and zero-shot experiments show poor results and demonstrate that the Iconclass retrieval remains a challenging task. We make our source code and models publicly available to support heritage institutions in the further enrichment of their digital collections.

Transfer Learning for the Visual Arts: The Multi-Modal Retrieval of Iconclass Codes

A Visual Analytical Approach for Transfer Learning in Classification.

Exploiting CLIP-based Multi-modal Approach for Artwork Classification and Retrieval

Multimodal Metadata Assignment for Cultural Heritage Artifacts

Object Retrieval and Localization in Large Art Collections using Deep Multi-Style Feature Fusion and Iterative Voting

Iconographic Image Captioning for Artworks

Visual Navigation of Digital Libraries: Retrieval and Classification of Images in the National Library of Norway's Digitised Book Collection

Challenging deep image descriptors for retrieval in heterogeneous iconographic collections

Learning Transferable Visual Models From Natural Language Supervision

Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritage

Towards Generating and Evaluating Iconographic Image Captions of Artworks

Adaptive Cross-Modal Prototypes for Cross-Domain Visual-Language Retrieval

An analysis of the transfer learning of convolutional neural networks for artistic images

K-LITE: Learning Transferable Visual Models with External Knowledge

Multimodal Semantic Transfer from Text to Image. Fine-Grained Image Classification by Distributional Semantics

Heritage Iconographic Content Structuring: from Automatic Linking to Visual Validation

Towards a multimodal framework for remote sensing image change retrieval and captioning

Knowledge Transfer Across Modalities with Natural Language Supervision

Iclip: Bridging Image Classification and Contrastive Language-Image Pre-Training for Visual Recognition

Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer

Evolving Interpretable Visual Classifiers with Large Language Models